Word count: 517

Summary of the Reading

This paper seeks to help make it faster and easier for teams to find their ideal team structure. While many services allow teams to test out many different team structures to find the best one, many of those services can take a lot of time and can greatly affect the people who work on the team. Often times they have to switch structures so often that it makes it hard for the teams to concentrate on getting work done. 

The method proposed in the paper seemed to be very successful. It resulted in teams that were 38-46% more effective. The system works by testing different team structures and taking automatically generated feedback information (like performance metrics) to figure out how effective each structure is. It will then base its future combinations on this feedback. Each time a new structure is tested, it varies on a five dimensions: hierarchy, interaction patterns, norms of engagement, decision-making norms, and feedback norms.

Reflections and Connections

I think that this paper has an excellent idea for a system that can help teams to work better together. One of the most important things about a team is how it is structured. The structure of a team can make or break its effectiveness, so getting the structure right is very important to making an effective team. A tool like this that can help a team figure out the best structure with minimal interruption will be very useful to everyone in the business world who needs to manage a team. 

I also thought that it was a great idea to integrate the system into Slack. When I worked in industry last summer, all of the teams at my company used Slack. So, it makes a lot of sense to implement this new system in a system that people are already familiar with.  The use of Slack also allows the creators to make the system more friendly. I think it is much better to get feedback from a human-like Slack bot than some other heartless computer program. It is also very cool how the team members can interact with the bot in Slack. 

I also found the dimensions that they used in the team structures to be interesting. It is valuable to be able to classify teams in some concrete way based on certain dimensions of how they perform. This also has a lot of real world applications. I think that a lot of the time, one of the hardest things in any problem space is just to quantify the possible states of the system. They did this very nicely with the team dimensions and all of their values. 

Questions

1. Would you recommend this system to your boss at your next job as a way to figure out how to organize the team?
2. Aside from the ones listed in the paper, what do you think could be some limitations of the current system?
3. Do you think that the possible structures had enough dimensions and values for each dimension?

Read More

Summary

The authors mention that the previous HCI research focused on ideal team structures and how roles, norms, and interaction patterns are influenced by systems. The state of research directed teams towards those structures by increasing shared awareness, adding channels of communications, convening effective collaborators. Yet organizational behavior research denies the existence of universally ideal team structures. And believes that structural contingency theory has demonstrated that the best team structures depend on the task, the members, and some other factors. The authors introduce Dream Team, a system that identifies effective team structures for each team by adapting teams to different structures and evaluating each fit. Dream Team explores over time, experimenting with values along many dimensions of team structures such as hierarchy, interaction patterns, and norms. The system utilizes feedback, such as team performance or satisfaction, to iteratively identify the team structures that best fit each team. It helps teams in identifying the structures that are most effective for them by experimenting with different structures over time on multi-armed bandits.

Reflection

The paper presented a system that focuses on virtual teams. In my opinion, the presented system is a very specific application to a very specific problem. The authors address their long list of limitations, including how they don’t believe their system generalizes to other problems easily. I also believe that the way they utilize feedback in the system is complex and unclear. Their reward function did not explain how qualitative factors were taken into consideration. The authors mention that high variance tasks would require more time for DreamTeam to converge.

Which means more time to get a response from the system, and I don’t know how that would be useful if it slows teams down? Also, when looking at the snapshot of the slack integration, it seems that they handle team satisfaction based on users response to a task, which is not always the case when it comes to collaboration on slack. The enthusiasm of the responses just seems out of the norm. The authors did not address how would their system address “team satisfaction” when there’s little to no response? Would that be counted as a negative response? Or would it be neutral? And even though their system worked well for the very specific task they chose, it’s also a virtual team. Which raises questions about how would this method be applicable for in person teams or hybrid teams? It seems that their controlled environment was very controlled. Even though they presented a good idea, I doubt how applicable it is to real life situations.

Discussion

• In your opinion, what makes a dream team?
• Are you pro or against ideal team structures? Why?
• What were the qualities of collaborators in the best group project/research you had?
• What makes the “chemistry” between team members?
• What does a successful collaborative team project look like during a cycle?
• What tools do you use in project management? 
• Would you use DreamTeam in your project?
• What would you change in DreamTeam to make it work better for you?

Read More

How do you identify the best way to structure your team? What kind of leadership setup should it have, how should team members collaborate and make decisions? What kind of communication norms should they follow? These are all important questions to ask when setting up a team but answering them is hard, because there is no right answer. Every team is different as a function of its team members. So it is necessary to iterate on these dimensions and experiment with different choices to try and see which setup works best for a particular team. Earlier work in CSCW attempts this with “multi-arm bandits” where each dimension is independently experimented with by a so called “bandit” (a computational decision maker) in order to collectively reach a configuration based on recommendations from each bandit for each dimension. However, this earlier work suffered from the problem of recommending too many changes and overwhelming the teams involved. Thus this paper proposes a version with temporal constraints, that still provides the same benefits of exploration and experimentation while limiting how often changes are recommended to avoid overwhelming the team.

This is my first exposure to this kind of CSCW literature and I find it a very interesting look into how computational decision makers can help make better teams. The idea of a computational agent looking at performance of teams and how they’re functioning and make recommendations to improve the team dynamics intuitively makes sense, because the team members themselves either can’t take an objective view because of their bias, or could be afraid to make recommendations or propose experimentation for fear of upsetting the team dynamic. The fact that this proposal is about incorporating temporal constraints to these systems is also a cool idea because of course humans can’t deal with frequent change because that would be very overwhelming Having an external arbiter do that job instead is very useful. I wonder whether if the failure of the human manager to experiment is because humans in general are risk averse, or the managers that were picked were particularly risk averse. This ties into my next complaint about the experiment sizes; both in the manager section and in the overall, I find the experiment size is awfully small. I feel like you can’t capture proper trends, especially socialogical trends such as being discussed in this paper, with experiments with just 10 teams. I feel a larger experiment should have been done to identify larger trends before this paper was published. Assuming that the related earlier work with multi-arm bandits also had similar experiment sizes, they should have been larger experiments as well before they were published.

1. could we expand the dreamteam recommendations where, in addition to recommending changes in the different dimensions, it is also able to recommend more specific things. The main thing I was thinking of was if it is changing heirarchy to a leader based setup, it also recommends a leader, or explicitly recommends people vote on a leader, rather than just saying “hey, you guys now need to work with a leader type setup”?
2. Considering how limited the feedback that DreamTeam could get, what else could be added than just looking at the scores at different time steps?
3. What would it take for managerial setups to be less loss averse? Is the point of creating something like DreamTeam to help and push managers to have more confidence in instituting change, or is it to just have a robot take care of everything, sans managers at all?

Read More

Summary:

The paper pointed out the theory that there is no universally ideas structures for effective teamwork. The best structure for teamwork is determined by different team members, tasks, surroundings, etc. Thus, this paper presents a system that investigates the optimal team structure by adapting different team structures to teams and evaluate the efficiency based on team performance and teamwork feedback. However, the combination of diverse dimensions of team structures and the arms (the different values of dimension) of each dimension is a large set. To avoid overwhelming group testers with these values, the paper also leverages a model called multi-armed bandits with temporal constrains which set the constraints of the number of arms selections based on several factors. The paper tested the platform with AMT workers, and evaluate the performance of the system with designed task and performance evaluation. The system confirmed that there are no two teams had the same optimal team structures, and this structure even different for the same group when completing the different tasks. The results also indicate the platform DreamTeam can promote high-efficiency teamwork.

Reflection:

First, I highly agree with the opinion that there are no universal idea structures for effective teamwork. Besides, searching the optimal structure through adapt different dimensions and evaluating each fit seems also reasonable. However, I think the experiment could gather more valuable information if they test the platform with a real group instead of a randomly formed group. Because I think the premise of becoming a group and completing a task together is that the group members are familiar with each other. Thus, this platform should be more effective in the early stages of team formation. Before the team members are familiar with each other, they can use this system to find the optimal team structure temporarily, so that they can quickly cooperate and work as a team even though they do not know each other. Nevertheless, as the familiarity among the group members raises, using this method to determine the optimal structure may be inefficient. Because they may have already find the best structure for some of the dimensions as they get along and getting more experience of working together.

I also considered another situation that is more suitable for this system, when a long-established team is assigned to working on a new type of task. Then maybe the working mode of the teamwork needs to be switched so that they can complete the new task most efficiently. At this time, the system support is demanded to find this new optimal structure.

Finally, I think the constraints method mentioned in the article also very inspirational. Maybe we can improve the effectiveness of the DreamTeam platform by allowing users to pre-delete some dimensions that they would not like to change. For example, the hierarchy or interaction pattern. In this case, the reduction of the combination is more conducive to exhaustive testing, and the adapting structure should be more fit for the teamwork.

Question:

1. What do you think using the computation power to decide the optimal structure for teamwork?
2. In this paper, the author finds random tester to form a group and complete the task, do you think this will influence the results?
3. Under what condition do you think this platform would most benefit.

Word Count: 544

Read More

Summary

Zhou et al.’s paper “In Search of the Dream Team” introduces DreamTeam — a system that identifies effective team structures for each group of individuals by suggesting different structures and evaluating the fit of each team. How team works relates with team structures, including roles, norms, and interaction patterns. Prior organizational behavior research doubts the existence of universally perfect structures. The rationale is simple: teams broast of great diversity so one single structure cannot satisfy the full functioning of each team. The proposed DreamTeam explores values along five dimensions of team structures including hierarchy, interaction patterns, norms of engagement, decision-making norms, and feedback norms. The system leverages feedback, randomly choosing metrics such as team performance or satisfaction, to iteratively identify the team structures that facilitate the best organization of each team. Also, the authors design multi-armed bandits with temporal constraints, an algorithm that determines the timing of exploration and exploitation trade-offs across multiple dimensions to avoid overwhelming teams with too many changes. In the experiments, DreamTeam is integrated with the chat platform Slack and achieves better performance and more diverse team structures compared with baseline methods.

Reflection

The authors design a system to facilitate the organization of virtual teams. Along with the several limitations mentioned in the paper, I feel the proposed DreamTeam system is based on a comparatively narrow scope of what makes a dream team and it seems difficult to generalize the framework to a variety of domains or platforms. 

In the first place, I do not agree that there is a universal approach to design or evaluate a so-called dream team. The components that make a dream team vary in different domains. For example, in sports, I would say personality and talent play important roles in forming a dream team. Actually, it goes beyond the term “forming” that a bunch of talented individuals not only bring technical expertise to the team, but they also contribute passion, strong work ethic, and strive for peak performance in the pursuit of excellence. To extend that point, working with people having similar personalities, similar values, similar pursuits will bring some chemistry to the team work which potentially enables challenging problem solving and strategic planning. All of these are not mentioned in the demensions and nearly impossible to be evaluated quantitatively. 

Also, I think it is important to make every team member understand their role, such as why they need to tackle the tasks and how that ties to a larger purpose beyond self’s needs. This provides a clear purpose and direction of where a group of people need to move forwards as a team. I do not think the authors emphasize the importance of how such understanding influences team member level of commitment. In addition, this kind of unified purpose can avoid duplication of member efforts and prevents pulling the efforts in multiple directions. 

Last but not least, in my opinion, basing on the maximizing of rewards is not the ideal way to determine the best team structures. Human society treasure process as well as results. It can be seen as a successful teamwork as long as the whole team is motivated and working on it. If too much emphasis is put on results, then the joy will be drained out of the job for the team. As long as progressive steps are made towards achieving the goal within a reasonable time frame, the team will become better. Building an ambitious, driven and passionate team is just the start. We need to ensure that the team members survive and are nurtured so that they can deliver on the targets.

Discussion

I think the following questions are worthy of further discussion.

• If you are the CEO of a company or a university president, would you consider using the proposed DreamTeam system to help organize your own team? Why or why not?
• Do you think the five bandits capture all dimensions to make a dream team?  
• Do you think the proposed DreamTeam system can be generalized to various domains? Are there any domains you think the system would not contribute towards an efficient team structure?
• Is there anything you can think about to improve the proposed DreamTeam system?

Read More

Summary:

The paper aims to find a Dream Team by adopting teams to different structures and subsequent evaluation. The authors try to identify the ideal team structure using “the multi-armed bandit” approach over time. The dream team structure selects the next exploration task based on the reward from the previous job. They explored a lot of background research on HCI groups, the structural contingency theory from organizational behavior, multi-armed bandit. A network of five bandits was created with different dimensions, namely, hierarchy, interaction patterns, norms of engagement, decision-making norms, and feedback norms. Each of the dimensions has different possible values. For example, for hierarchy, there can be three possible values – none, centralized (where a leader was elected), decentralized (majority vote). Global temporal constraint and dimensional temporal constraint are taken into consideration to determine at what stage the teams are prepared to embrace changes and also take into account if too many dimensions change at one. The authors used the popular game Codenames for the Slack interface. They used Amazon Mechanical Turk to employ 135 workers and assigned them based on five conditions, namely, control, collectively chosen, manager chosen, bandit chosen, and Dream Team is chosen. There were 35 teams with seven teams per condition. It was found that Dream Team based teams outperformed other teams 

Reflection 

The paper was a nice read on selecting the ideal team structure to maximize productivity. The paper did extensive background research on team structures and included theories from HCI and organizational behavior. Being from a CS background, I have no idea about what team structure is and the theory involved behind selecting the ideal structure. It was a very new concept for me, and the difference between the approaches taken by the HCI domain and Organizational behavior was intriguing. The authors described their approach in detail and mathematically, which makes it easy to visualize the problem as well as the method.

The most interesting section was the integration with Slack, where the Slack bot was utilized to guide the Team with broadcast messages. It was interesting to see how different teams reacted to the messages the Slack bot posted. Dream Teams mostly adhered to the suggestions of the Slack bot whereas, some of the other team structures chose to ignore them. It would be good if the evaluation is also done on a different task. The game is relatively simple, and we don’t know how the Dream Team structure would perform for complicated tasks. It would be intriguing to see how this work could be potentially extended.

The paper highlights a probabilistic approach to proposing the ideal team structure. One thing that was not very clear to me is how the slack bots suggest other than taking into consideration the current score and the best approach. Is it using NLP techniques to deduce the sentiment of the comment and then posting a cross-comment? 

Question

1. The authors used slack to test their hypothesis. How would dream-team perform for real-life software development teams?
2. The test subjects were Amazon Mechanical Turks, and the project was reasonably simple (codenames game). Would Dream Team performs better than the other structures when it is domain-specific, and experts are involved? Would it lead to more conflicts?
3. Could we use better NLP techniques and sentiment analysis to guide the DreamTeams better?

Read More

Paper:  Sharon Zhou, Melissa Valentine, and Michael S. Bernstein. 2018. In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 1–13. https://doi.org/10.1145/3173574.3173682

Summary: This paper introduces a system called DreamTeam that explores a search space for the optimal design of teams in an online setting. The system does this through multi-armed bandits with temporal constraints, a type of algorithm that manages the timing of exploration–exploitation trade-offs across multiple bandits simultaneously. This answers a classic question in HCI and CSCW: when should teams favor one approach over another? The paper contributes a computationally identifiable method of good team structures, a system that manifests this, and an evaluation with improvements of 46%. The paper concludes with a discussion of computational partners for improving group work, such as aiding us by pointing out our biases and inherent limitations, and helping us replan when the environment shifts.

Reflection:

I appreciate the way they evaluated the system and conducted randomized controlled trials for each of their experimental conditions. The evaluation is done on a collaborative intellective task, and I wonder how different the findings would be if they had evaluated it using a creative task, instead of an intellective or analytic task. Perhaps there is a different optimal “dream team” based not only on the people but the task itself. 

I also appreciate the thorough system description and how the system was integrated within Slack as opposed to having it be its own standalone system. This increases the real world generalizability of the system and also means that it is easier for others to build on top of. In addition, hiring workers in real-time would have been hard, and it’s unclear how synchronous/asynchronous the study was.

One interesting approach is considering both types of bandits simultaneously, exploration and exploitation. I wonder how the system might have fared if teams were given the choice to explore each on their own—probably worse. 

Another interesting finding is the evaluation with strangers on MTurk. I wonder if the results would have differed if it was a) in a co-located setting and/or b) among coworkers who already knew each other. 

Finally, the paper is nearly two years old, and I don’t see any follow up work evaluating this system in the wild. I wonder why or why not. Perhaps there is not much to gain through an in-the-wild evaluation, or that an in-the-wild evaluation did not fare well. Either way, it would be interesting to read about the results—good or bad.

Questions:

1. Have you thought about building a Slack integration for your project instead of a standalone system?
2. How might this system function differently if it were for a creative task such as movie animation?
3. How would you evaluate such a system in the wild?

Read More