13.4. Processing expert assessments

The processing of expert assessments during a group examination has specifics depending on the nature of the information expressing the preferences of experts and the substantive justification of their preferences, goals, purpose and other factors of the examination carried out and consists of the following:

determining a generalized assessment of the objects under study or the object in question according to a number of properties, indicators and their relative importance;

assessing the consistency and dependence of expert opinions;

assessing the reliability of the obtained calculated values.

The purpose of processing expert assessments is to obtain generalized data on the objects under study, the analysis of which allows us to obtain Additional information about the features of the assessment process, which allows us to formulate conclusions about the quality of the examination carried out and the reasons for possible differences in opinions of coalitions of experts.

The determination of a generalized assessment of the objects under study is carried out during a group expert assessment based on the use of methods of averaging individual assessments of experts, taking into account the assumption that they are sufficiently accurate “measurers” and their assessments form one or several compact groups. Algorithms for obtaining a generalized assessment depend on the types of methods used for subjective measurement by experts of the preference of the objects being assessed or their properties. If the results of the applied subjective measurement methods are numbers or points, then the construction of a group assessment consists of determining the average value (mathematical expectation) or median (the most probable estimate). In another case, if the results are ranks, then the processing task is to construct a generalized ranking of objects based on the best way to harmonize the individual rankings of experts in the form of a median, the sum of the distances from which the results of individual rankings is minimal.

By ordering the obtained results of generalized assessments of objects in descending order of their importance, one can judge their relative importance. Additional indicators that clarify the relative importance of the objects under study are: the frequency of the highest (maximum possible) ratings for the object, the sum of the object’s ranks. Frequency of maximum possible estimates for j-object is determined by the formula:

where is the number of maximum possible ratings received j-th object;

–number of experts assessing j th object of study.

It is advisable to use this indicator to establish the order of objects in the case of obtaining equal values ​​of the results of generalized assessments.

The sum of the ranks of the research object is determined by the formula:

where is the rank of the assessment j th expert j- th object.

If among the assessments of the j-m data by the expert there are identical ones, then they are assigned the same rank, equal to the arithmetic mean of the corresponding numbers in the natural series. When assessing the relative importance of objects, the object characterized by its lowest value should be considered the most important.

A quantitative assessment of the consistency of expert opinions is necessary if the experts’ opinions differ on the objects under consideration for a more reasonable interpretation of their discrepancies. At the same time, individual assessments of the object in question, expressed by experts, are represented as points in a certain space in which there is a concept of distance. Using the concept of compactness, we can interpret the degree of agreement between experts’ opinions; then, if the indicated estimates are located at a short distance from each other, forming a compact group, then we can talk about good agreement between experts’ opinions, otherwise – low agreement. If expert assessments form two or more compact groups in space, this means that in the expert group there are corresponding coalitions with significantly different points of view on the assessment of objects. The variety of methods proposed in the literature for assessing the consistency of expert opinions is determined by the use of various subjective measurement methods to evaluate objects, the results of which can be numbers, scores or ranks, as well as various measures of the degree of agreement (for example, a measure of the consistency of expert assessments can be the ratio of the standard deviation to the mathematical expectation random variable; the sum of the distances of estimates from the average value, related to the distance of the mathematical expectation from the origin; the number of points located within the radius of the standard deviation from the mathematical expectation to the entire number of points, etc.). Some methods for determining the consistency of quantitative estimates based on the concept of compactness are discussed in Section 11.4.

The following indicators of the degree of agreement between expert opinions are used: coefficient of variation, coefficient of paired rank correlation (Spearman or Kendall), coefficient of concordance (dispersion or entropy).

The coefficient of variation (V j ) estimates, data j- mu object is determined by the formula:

where is the score in points i th expert j-th object;

– the average statistical value of the object’s assessment in points, determined by the formula:

Where m j – number of experts assessing j- th an object.

The lower the value of this coefficient, the higher the degree of agreement between expert opinions.

Spearman's paired rank correlation coefficient for two expertsα and β is determined by

where are ranking scores j- th objects of experts α and β;

P– number of objects being assessed;

– indicators of related (equal) ranks of expert assessments α and β , calculated as follows:

If everyone n ranks of assessments assigned i-th expert are different, then T i= 0, otherwise for equal ranks:

Where L – number of groups of related ranks;

t 1 – number of related ranks in 1- th . group.

The value of the coefficient indicates complete agreement between expert opinions α and β; meaning - about the complete opposite of expert opinions; meaning – about the lack of connection between the opinions of experts.

To assess the degree of agreement between the opinions of the entire group of experts as a whole, the concordance coefficient is used. The concordance coefficient is determined in the following sequence: first, the arithmetic mean of the sums of the ranks of ratings of all objects is calculated:

then the deviations are calculated d j sum of ranks of assessments received j- m object from:

after this the indicators are calculated T i related (equal) ranks of ranking scores assigned i-th expert; Finally, the concordance coefficient is calculated:

Where m 1 – the number of experts who assessed at least one object.

The concordance coefficient varies from 0 to 1. An increase in the value of the concordance coefficient corresponds to an increase in the degree of agreement between expert opinions. The small value of the concordance coefficient may be due either to a truly low degree of agreement between experts’ opinions, or to the existence of groups with a high agreement of opposing opinions.

Assessments of objects obtained as a result of processing expert assessments are random variables. Therefore, it is necessary to evaluate the reliability (reliability, level of significance) of the examination results. To determine the level of significance, the so-called chi-square goodness-of-fit test is used. The sequence for determining the significance level for this criterion is as follows:

the value is calculated using the formula:

Where T– number of experts,

then the number of degrees of freedom is calculated ( r = n– 1, where n– number of objects under study).

According to the table of values ​​for a certain number degrees of freedom and the found value, the probability is determined R random occurrence of the calculated value of the opinion consistency indicator. Then a certain threshold probability value is fixed - P o (usually P o = 0.05 or 0.01), called the significance level. If R turns out to be less than P o , then the hypothesis about the random origin of a particular value of the agreement of opinions indicator is rejected, that is, this indicator is considered significant, and the group of experts is considered representative. In another case, if the hypothesis about the random origin of a particular value of the agreement indicator is accepted, then this indicator is considered insignificant, and the group of experts is considered unrepresentative.

Let's consider an example of using expert assessments to determine the impact of integrated automated information management systems (IAIS) on cost items of the cost of products manufactured by a manufacturing enterprise.

As practice shows, experts who design IAISU, as well as a group of specialists who operate this system, should participate as experts. Before the start of the examination, all its participants receive initial information about the implemented local AIMS and a list of cost items that they can influence in the form of a table, where the list of cost items is located horizontally, and the implemented local AIMS is located vertically. There must be at least four experts from development specialists. The head of the department can act as experts i-th local AISU, leading development specialist i-th local AIMS (task, complex of AIMS organizational management), economist of the AIMS department, etc. In turn, there must be at least six experts from specialists involved in operating the system.

The quality of expert assessments, their reliability and validity largely depends on the chosen methodology for collecting and processing expert opinions. The customized method we use to identify impact i-x local AIMS for cost items of product costs, includes conducting a questionnaire survey, selecting and processing the received conclusions. In this case, the information array consists of tables (questionnaires) of expert assessments filled out by specialists. In relation to solving our question, we use the method of expert assessments outlined above and in the work. When compiling tables of expert assessments, three conditions must be met:

quantitatively determined answers to the proposed questions were obtained;

formalized information was obtained about the nature of the sources of argumentation, as well as the degree of influence of each source on the expert’s answer;

obtained from experts quantified estimates of the degree of their familiarity with the area to which the proposed questions relate.

In order to satisfy the first condition, the questions should be reduced to assessing the relative importance of the influence of 1 local AIMS on cost items of product costs. Each expert is asked to give an assessment (using a hundred-point system) of the relative importance of the influence of the specified AIMS on the cost items of the cost of production. A questionnaire in the form of a table (Table 10, p. 298) is issued to each expert, which contains information vertically about the list of designed tasks (complexes), local AIMS, and horizontally a list of cost items of production costs, some of which they can influence .

Table 9

There are often many alternatives to choose from, each with different benefits. And how can you choose the best one, having the opinion of dozens, or even hundreds of experts?

How to calculate the rating computer game, based on critics’ assessments of graphics, gameplay and plot, and the collective choice of a priority task before the appearance of the customer, refers to the methods of expert assessments.

Brief educational program

In cases of extreme complexity of the problem, its novelty, insufficiency of available information, and the impossibility of mathematical formalization of the solution process, one has to turn to the recommendations of competent specialists who know the problem perfectly well—experts. Their solution to the problem, argumentation, formation of quantitative estimates, processing of the latter by formal methods are called the method of expert assessments.

1. Individual assessments are based on the use of the opinions of individual experts, independent of each other.
2. Collective assessments are based on the use of collective expert opinion.

Ways to measure objects

1. Ranging- this is the arrangement of objects in ascending or descending order of some inherent property. Ranking allows you to select the most significant factor from the set of factors being studied.
2. Paired comparison- this is the establishment of preference for objects when comparing all possible pairs. Here, as in ranking, there is no need to order all objects; it is necessary to identify a more significant object in each pair or establish their equality.
3. Direct assessment. It is often desirable not only to order (rank the objects of analysis), but also to determine how much more significant one factor is than others. In this case, the range of changes in the characteristics of an object is divided into separate intervals, each of which is assigned a certain score (score), for example, from 0 to 10. That is why the direct assessment method is sometimes also called the point method.

Method for setting weight coefficients (a ij)

1. all characteristics are assigned weighting coefficients so that the sum of the coefficients is equal to some fixed number (for example, one, ten or one hundred);
2. the most important of all features is given a weighting coefficient equal to some fixed number, and all the others are given coefficients equal to fractions of this number.

1. the expert arranges all features in decreasing order of their importance: A1>A2>…>An;
2. assigns a value equal to one to the first characteristic: A1=1, and assigns weighting coefficients to the remaining characteristics in fractions of one;
3. compares the value of the first attribute with the sum of all subsequent ones.

IN pairwise comparison There is no need, as in ranking, to order all objects; it is necessary to identify a more significant object in each pair or establish their equality. Paired comparisons can be made when large number objects, as well as in cases where the difference between objects is so insignificant that their ranking is practically impossible.
When using the method, a matrix of size n x n is most often compiled, where n is the number of objects being compared.

When comparing objects, the matrix is ​​filled with elements a ij as follows (another filling scheme can be proposed):

• 2, if object i is preferable to object j (i > j),
• 1, if equality of objects is established (i = j),
• 0 if object j is preferable to object i (i< j).

And now, the best part...

Analysis of the results of expert assessments

Formation of a generalized assessment

Moreover, the result may consist of several algorithms, intertwined with others. For example, the algorithm for calculating the expert’s competence coefficient can influence the average statistical assessment of this expert, etc.

Establishing the degree of consistency of expert opinions

Average linear deviation

Standard deviation

Dispersion

Spearman's rank correlation coefficient

The coefficient (value) can vary in the range from –1 to +1. If the estimates are completely identical, the coefficient is equal to one. The coefficient is equal to minus one when there is the greatest discrepancy in expert opinions.
x ij – rank ( importance), assigned to the i-th object by the j-th expert, x ik is the rank assigned to the i-th object by the k-th expert, d i is the difference between the ranks assigned to the i-th object.

Kendell's coefficient of concordance
The coefficient can take values ​​ranging from 0 to 1. With complete agreement of expert opinions, the concordance coefficient is equal to one, while complete disagreement is zero. The most realistic case is the case of partial agreement of expert opinions.

Calculation

The average rank of the set of characteristics is determined:

The deviation d j of the average rank of the j-th characteristic from the average rank of the population is calculated:

The number of identical ranks assigned by experts to the j-th attribute – t q – is determined.
The number of groups of identical ranks is determined - Q. The concordance coefficient is determined by the formula:

Where

Speaking about the consistency of expert opinions, it is worth mentioning that ranking does not (or does not always imply) distance. That is, for one expert, A>B>C means that A>>B>C, and for another, A>B>>C. And all sorts of correlations and calculations of average ratings will not help here. Alternatively, consider the consistency index. Something like the number of contradictory closed chains of expert opinions (the first believes that A is better than B, the second that B is better than C, and the third that C is better than A) to the number of all such chains.

Conclusion

So I take my leave. Happy holiday everyone, Ramin. And for those who came to look at the girls - here you go

In most of the types of research considered, the subject of study is the point of view of consumers (private or corporate). However, there are also studies that are carried out taking into account all the factors that influence a particular market - competition, affiliation, general trends, changes in legislation, current and planned projects of players, industry regulation, risks, etc. And neither These studies are not included in publications or industry statistics. Here, both desk research and consumer surveys can be used as an element, but the main tool in this case is expert interviews with market players, independent analysts, association leaders, journalists, people involved in competitive intelligence, etc.

Expert assessment method - this is a type of research in which the respondents are experts - specialists in a certain field of activity.

The main purpose of the expert assessment method - identifying complex aspects of the problem under study, increasing the reliability of information and conclusions.

Distinctive feature of the method is that it presupposes the competent participation of experts (expertise) in the analysis and solution of research problems.

Expertise - procedure for obtaining information from experts. Expert assessments are expert judgments about various spheres of human activity, involving a procedure for comparing objects and their properties according to selected criteria.

Specifics of the expert survey is as follows.

There is no need to use indirect or control questions in the questionnaire.

The expert survey program is not detailed and is of a conceptual nature.

In the questionnaire, it is preferable to use open-ended questions with complete freedom to choose the form of the answer.

Basic regulatory requirements for conducting expert assessments :

careful selection of experts;

assessing the reliability of information provided by experts;

creating conditions for the productive use of experts during the research;

taking into account factors influencing expert judgments;

preservation of expert information without distortion at all stages of the study.

The quality and reliability of expert assessments is reduced by the disorderly selection of experts.

Criteria for selecting experts are:

degree of competence, indicators of which can be the presence of an expert’s academic degree, academic title, work experience in the specialty, official position, number of published works, etc.;

ability to navigate the latest developments modern science in those areas that are the subject of expertise;

a combination of narrow specialization and the general outlook of an expert;

ability to analyze and synthesize the problems being studied,

the ability to process and assimilate qualitatively new information;

high moral qualities;

a combination of psychologically acceptable to each other in a group of experts of different ages, different scientific schools etc.

The expert group cannot be large. Expert selection methods let's highlight objective - use of special selection techniques - and subjective - involvement of potential experts themselves in the selection procedure.

The objective approach has two options :

a) documentary method - selection of experts based on socio-demographic data.

b) experimental method - selection based on candidate testing.

The subjective approach also has several variations ;

a) certification - selection of experts is carried out using open or secret voting of potential members of the future expert group (can be carried out in several rounds);

b) method of mutual assessment in points or ranking;

c) method of self-assessment of competence.

An expert survey can take different forms :

one-time individual survey (questionnaire or interview);

one-time collective survey (meetings, brainstorming);

individual survey in several rounds (Delphic technique);

collective survey in several rounds (discussion, meeting, multi-stage selection).

Another form of expert survey is traditional discussion - a method of conversation with a small group of experts on a research topic. The purpose of the discussion is to develop a group collective opinion. A prerequisite for a successful group discussion is that the subject of the discussion is clearly stated and that the vast majority of participants are familiar with it.

To a large extent, the success of the discussion depends on the culture, preparation, conduct and presentation of the collective opinion of experts.

"Referred Valuation" Method - repetition of several cycles of discussion with the identification of the essence of emerging disagreements and the gradual development of a common opinion of all or the majority of participants in the examination, while those who disagree retain the right to a private opinion.

Examination methods in the collective work of its participants, they have many obvious advantages, but at the same time they also have a number of disadvantages. The main disadvantage is related to the influence of experts on each other. This disadvantage is overcome by using an individual survey in several rounds. The correspondence version of the “referred assessment” method acquired the name Delphi method, or the Delphic technique (from the name of the ancient Greek city, which became famous as a center of oracle predictions).

The Delphi technique guarantees the anonymity of respondents: experts do not meet each other; they fill out anonymous questionnaires or connect directly to work with a computer.

After the first round, the experts become familiar with the final characteristics of the position of the group as a whole. In the second round, they have the opportunity to either bring their opinion closer to the position of the majority, or to study the reason for the deviation. In the third round, new information opens up the opportunity to reconsider your point of view.

Simplified variations of the Delphi technique (“mini Delphi”) allow

collect expert assessments for 2-3 rounds in a few hours or days.

Disadvantages of the Delphi method include :

complexity of preparation, implementation and processing of results,

relatively large investment of time and money.

Despite its shortcomings, the Delphic technique has become widespread; In terms of the scale of application in many countries, it is among the top five most popular methods of social forecasting.

Expert survey in modern form often based on the combined use of various methods, forms and procedures. Thus, based on the use of the Delphic technique, one of the American forecasting systems, “Pattern,” was built, which allows you to create a system of information models in the form of a tree of goals. Work on creating a system begins with creating a scenario, i.e., a description of the state and direction of development of the object under study. At the next stage, a tree of goals is built; for each goal, necessary and sufficient subgoals are developed, which are a condition for achieving the overall goal. At the third stage, the coefficients of the relative importance of criteria and goals at all levels are determined. Next are determined specific types necessary work, resources and timing for their implementation. The longest chain is the time it takes to complete the entire work package.

When analyzing the collected expert data in accordance with the objectives of the study and the accepted models, it is necessary to present the information received from experts in a form convenient for decision-making (arrange objects - options, indicators, factors, etc.), and also determine agreed - the validity of expert actions and the reliability of expert assessments.
For example, identified in the process qualitative analysis risks must be presented in order of their importance (the degree of their possible impact on the level of losses), or risk reduction options - in order of their preference, etc.
There are a number of ordering methods, each with its own advantages and disadvantages, as well as scope. effective application. The most common of them are: ranking, direct assessment, sequential comparison, paired comparison.
An important point in expert procedures is the assessment of the consistency of experts’ actions and the reliability of expert assessments.
As noted, existing methods for determining the reliability of expert assessments are based on the assumption that if the actions of experts are coordinated, the reliability of the assessments is guaranteed.
Most often, for these purposes, the coefficient of concordance (agreement) is used, the value of which allows us to judge the degree of consistency of expert opinions and, as a consequence, the reliability of their assessments.
The value of the concordance coefficient can vary from 0 to 1. At W = 0 there is no concordance, i.e. There is no connection between the assessments of different experts. When W = 1, the agreement between experts’ opinions is complete.
To make a decision on the use of estimates obtained from experts, it is necessary that the concordance coefficient be greater than the given (normative) value Wн (W > Wн).
You can take Wн = 0.5. It is believed that when W > 0.5 the actions of experts in to a greater extent agreed upon rather than not agreed upon.
Let us consider the determination of the concordance coefficient using the following simplified example. Let the process of qualitative analysis identify five types of risks to which the project may be exposed during its implementation. Experts are faced with the task of ranking these risks (presented in order of their importance) according to the degree of their possible impact on the level of losses.
With a completely agreed upon opinion of the experts, each of them gave two points to the first type of risk, one to the second, four to the third, three to the fourth, and five to the fifth. In this case, the total rank of risk importance would be 8, 4, 16, 12 and 20 points, respectively.
It is known that the average value of the total assessment for m objects assigned by n experts is 1/2 n (m+1) and in the example under consideration the average value is 12.
Thus, the opinions of experts can be considered sufficiently consistent.
Criteria are also used to assess the likelihood that the agreement between experts was not the result of random variations in their opinions.
If, in accordance with accepted criteria, expert opinions can be considered agreed upon, then their assessments are accepted and used in the process of preparation and implementation management decisions.
If the estimates obtained cannot be considered reliable, the survey should be repeated again. If this does not give the desired results, the initial data should be clarified and (or) the composition of the expert group should be changed.
Here it should be noted the important role of the organizers of the examination, whose tasks include:
formulation of the problem, determination of the goals and objectives of the examination, its boundaries and main stages;
development of examination procedures that best suit the nature of the problem being solved; selection of experts, verification of their competence and formation of expert groups;
conducting a survey and agreeing on assessments; formalization of the received information, its processing, analysis and interpretation.
The effectiveness of the results obtained through expert assessments, incl. and the reliability of these estimates.

More on the topic Analysis and processing of expert assessments:

1. Expert procedures and methods of subjective assessments when measuring risk

The main idea of ​​forecasting based on expert assessments is to build rational procedure for human intuitive-logical thinking in combination with quantitative methods for assessing and processing the results obtained.

The essence of expert assessment methods is that the forecast is based on opinion specialist or team of specialists, based on professional, scientific and practical experience.

Individual expert assessments- are based on the use of the opinions of expert specialists in the relevant profile.

1. Interview method involves a conversation between a forecaster and an expert using a question-and-answer scheme, during which the forecaster, in accordance with a pre-developed program, poses questions to the expert regarding the prospects for the development of the forecasted object. The success of such an assessment depends to a large extent on the ability of the expert to give impromptu opinions on a wide variety of issues.

2. Questionnaire method consists in the fact that the expert is asked to fill out a questionnaire (questionnaire) containing a list of questions, each of which is logically related to the research task.

The following types of questions can be used in the questionnaire:

open – answers to these questions can be formulated in any form;

closed type – answer options are offered, one of which must be chosen by the expert.

The use of closed-ended questions in the questionnaire is preferable, as it simplifies the statistical processing of the answer results and facilitates the work of the expert when filling out the questionnaire. On the other hand, the list of answers to the question may not contain the expert’s opinion. Therefore, when creating a list of answer options for some questions, it should be possible for the expert to put forward his own answer option or avoid answering.

3. Analytical method (analytical notes) provides for careful independent work by the expert to analyze trends, assess the state and development paths of the predicted object. The expert can use all the information he needs about the forecast object. He draws up his conclusions in the form of a memorandum. The main advantage of this method is the ability to make maximum use of the expert's individual abilities. However, it is of little use for predicting complex systems and developing strategies due to the limited knowledge of one specialist expert in related fields of knowledge.

The main advantage of individual expert assessment methods is the possibility of maximizing the use of the individual abilities of experts. However, these methods are not very suitable for predicting the most general strategies due to the limited knowledge of one expert about the development of related fields of science and practice.

An example of the use of expert assessments in planning the development of socio-economic systems is the multicriteria task of choosing a solution option, which is currently relevant in many areas of human activity.

The multicriteria selection procedure includes the following steps:

Identification of the most significant indicators (criteria) characterizing the object under study;

Determining a method for quantitative assessment of indicators;

Determination of acceptable limits for changes in indicators;

Choosing a method for finding the best option;

Solving the problem and analyzing the results.

As an objective function for evaluating solution options, additive convolution of criteria is most often used:

or
, (2.18)

Where - weighting coefficients characterizing the significance of the criterion . Numerical values determined by experts, and it is desirable to comply with the following conditions:

. (2.19)

If the criteria
have different units of measurement, they must be brought to a single dimensionless scale so that the following inequalities are satisfied:

(2.20)

(2.21)

Example . According to experts, the main indicators of economic and social development of the region are:

gross domestic (regional) product;

employment level;

average monthly salary.

An expert assessment of the significance of the criteria on a ten-point scale is presented in Table. 2.2.

The regional leadership has been proposed four targeted regional development programs aimed at priority financing:

Agro-industrial complex;

Food industry enterprises;

Branches of the socio-cultural sphere;

Housing construction.

The expected values ​​of the main indicators obtained during the implementation of the target programs under consideration are given in Table. 2.3.

Table 2.2

Expert assessment results

Table 2.3

Expected values ​​of the main socio-economic

regional development indicators

It is necessary to determine the most appropriate program for the development of the region.

Solution:

Let's determine the values ​​of the weighting coefficients:

;
;
.

Thus, as a result of processing expert assessments, the objective function has the following form:

Considering that target program No. 3 is obviously ineffective compared to program No. 2 (1500<2000; 80=80; 1000<2000), удалим её из матрицы возможных решений:

Since the values ​​of the indicators have different dimensions, they must be brought to a single dimensionless scale. This is achieved by dividing the elements of each column by the maximum value in the column:

At the final stage, we determine the value of the objective function for the proposed programs:

The maximum value of the objective function corresponds to program No. 1. Therefore, the implementation of this program is most appropriate.

The most reliable are collective expert assessments - involve determining the degree of consistency of expert opinions on promising areas of development of the forecast object, formulated by individual specialists.

To organize expert assessments, working groups are created whose functions include conducting a survey, processing materials and analyzing the results of a collective expert assessment. The working group appoints experts who provide answers to the questions raised regarding the prospects for the development of this facility.

1. The essence method of collective idea generation (brainstorming) consists of using the creative potential of specialists in brainstorming a problem situation, which first involves the generation of ideas, and then their structuring, analysis and criticism, putting forward counter-ideas and developing a coherent point of view.

The method of collective idea generation involves the implementation of the following stages:

1. forming a group of brainstorming participants to solve a specific problem. The optimal group size is found empirically. Groups consisting of 10-15 people are considered the most productive.

2. The analysis group draws up a problem note, which formulates the problem situation and contains a description of the method and the problem situation.

3. Idea generation stage. Each participant has the right to perform multiple times. Criticism of previous performances and skeptical remarks are not allowed. The facilitator adjusts the process, welcomes improvements or combinations of ideas, and provides support, freeing participants from constraint. The duration of the brainstorming session is no less than 20 minutes and no more than 1 hour, depending on the activity of the participants.

4. Systematization of ideas expressed at the generation stage. A list of ideas is formed, characteristics by which ideas can be combined are identified, ideas are combined into groups according to the identified characteristics.

5. At the fifth stage, the destructuring (destruction) of systematized ideas is carried out. Each idea is subject to comprehensive criticism by a group of highly qualified specialists consisting of 20-25 people.

6. At the sixth stage, critical comments are assessed and a list of practically implementable ideas is compiled.

Method "635" - one of the varieties of “brainstorming”. The numbers b, 3, 5 indicate 6 participants, each of whom must write down 3 ideas within 5 minutes. The leaf goes around in a circle. Thus, in half an hour, everyone will write down 18 ideas, and all together - 108. The structure of ideas is clearly defined. Modifications of the method are possible. This method is widely used in foreign countries (especially in Japan) to select from a variety of ideas the most original and progressive solutions to certain problems.

2. Delphi method. The purpose of the method is to develop a program of sequential multi-round individual surveys. Individual surveys of experts are usually carried out in the form of questionnaires. Then they are statistically processed on a computer and a collective opinion of the group is formed, arguments in favor of various judgments are identified and summarized. The computer-processed information is communicated to experts, who can adjust the assessments, while explaining the reasons for their disagreement with the collective judgment. This procedure can be repeated up to 3-4 times. As a result, the range of assessments is narrowed and a consistent judgment is developed regarding the prospects for the development of the object.

Features of the Delphi method:

a) anonymity of experts - interaction between group members when filling out questionnaires is completely excluded;

b) the possibility of using the results of the previous round of the survey;

c) statistical characteristics of group opinion.

3. "Commission" method - based on the work of special commissions. Groups of experts at a round table discuss a particular problem in order to harmonize points of view and develop a common opinion. The disadvantage of this method is that the group of experts in their judgments is guided mainly by the logic of compromise.

The method of expert commissions can be organized in one of the following forms:

As practice has shown, the “commission” method has significant disadvantages:

the great influence of such a psychological factor as the opinion of authoritative experts, to which other experts join without expressing their point of view;

the reluctance of experts to publicly renounce their previously expressed opinions;

During the work of commissions, a dispute most often occurs between two or three of the most authoritative experts, as a result of which other experts either do not take part in the discussion or their opinions are not taken into account.

4. Method of trial – is based on organizing the work of a team of experts in the form of conducting a trial. The use of this method is advisable when there are several groups of experts, each of which defends its own point of view. In this case, the object of forecasting acts as the “defendant”. Leaders of groups expressing alternative points of view act as prosecution and defense (prosecutor, lawyer). Individual experts play the role of witnesses, providing the court with the information necessary to make a decision. The role of the judge is played by an interested person (group of persons). For example, in the television program “The Trial,” based on the use of the court method to analyze and predict the development of various socio-economic processes, the role of the judge was played by the audience, voting during the program by telephone calls for the point of view that they supported.

Method of morphological analysis involves choosing the most acceptable solution to a problem from among the possible ones. It is advisable to use it when forecasting fundamental research. The method includes a number of techniques that involve a systematic consideration of the characteristics of an object. The study is carried out using the “morphological box” method, which is built in the form of a tree of goals or a matrix in which the corresponding parameters are entered in the cells. Connecting a parameter of the first level in series with one of the parameters of subsequent levels is a possible solution to the problem. The total number of possible solutions is equal to the product of the number of all parameters presented in the "box", taken by row. Through permutations and various combinations, it is possible to develop probabilistic characteristics of objects.

Script writing method- is based on determining the logic of a process or phenomenon over time under various conditions. It involves establishing a sequence of events that develop during the transition from the existing situation to the future state of the object. The forecast scenario determines the development strategy of the forecasted object. It should reflect the general goal of the development of the object, the criteria for assessing the upper levels of the “goal tree”, the priorities of problems and resources to achieve the main goals. The scenario displays a sequential solution to the problem and possible obstacles. In this case, the necessary materials for the development of the forecast object are used.

A forecast graph is a figure consisting of vertex points connected by edge segments. A “goal tree” is a tree graph expressing the relationship between vertices-stages or problems of achieving a goal. Each vertex represents a target for all branches emanating from it. The “tree of goals” involves identifying several structural or hierarchical levels.

Building a “tree of goals” requires solving many problems: forecasting the development of the object as a whole; formulating a scenario for the predicted goal, determining the levels and vertices of the “tree”, criteria and their weights in ranking the vertices. These tasks can be solved, if necessary, using expert assessment methods. It should be noted that this goal as a forecast object can correspond to many different scenarios.

The scenario is usually multivariate in nature and highlights three lines of behavior: optimistic - the development of the system in the most favorable situation; pessimistic - development of the system in the least favorable situation; working - development of the system taking into account counteraction to negative factors, the occurrence of which is most likely. As part of the forecast scenario, it is advisable to develop a backup strategy in case of unforeseen situations.

The finished script must be analyzed. Based on the analysis of information found to be suitable for the upcoming forecast, goals are formulated, criteria are determined, and alternative solutions are considered.