سیستم پشتیبانی از تصمیم گیری برای انتخاب فیلم انیمیشن بر اساس تجمع چند معیاری از اولویت های ترتیبی داوران

This paper presents a decision support system devoted to the selection of films for the International Animated Film Festival organized at Annecy, France, every year. It deals with the representation and aggregation of referees’ preferences along predefined criteria in addition to their overall selection point of view. The practical requirements associated with this application (often encountered in social or cultural areas as well) are: a common ordinal scale for the criteria scores, a procedure to deal with inconsistencies between criteria and overall scores, explanation tools of each referee’s preference model in order to facilitate the deliberation process and also to argument the selection decision. The processing of referees’ preferences is achieved thanks to a recent method which consists in finding a generalized mean aggregation operator representing the preferences of a referee, in a finite ordinal scale context. The method allows to deal with consistency conditions on referees’ behaviour in order to highlight the criteria or pair of criteria which are the most influential for each of the referees. All the functionalities have been implemented in an interactive decision software that facilitates a shared selection decision. Results issued from the 2007 selection are presented and analysed from the preference representation and processing point of view.

The real-sized application considered in this paper is the selection of films for the International Animated Film Festival organized every year at Annecy, France. The board decision making appointed by the festival organizing committee is based on the three referees’ preferences along predefined criteria (Scenario, Aesthetics, Animation Technics and Soundtrack) in addition to their overall selection point of view (yes, perhaps, no) on each of the 600 films they watch in one week (100 films per day). The practical requirements associated with this application, but encountered also in other applications such as student evaluation or scientific paper selection, are: •a common ordinal scale for the criteria scores (e.g. bad, medium, good), avoiding any numerical conversion in order to faithfully represent the revealed preferences of the referees, •a procedure to deal with inconsistencies between criteria and overall scores due to the limited accuracy and reliability of referees or generated by the very demanding conditions of the evaluation task, •explanation functionalities of each referee preference model, as the three models are generally conflicting, in order to facilitate the deliberation process and also to argue the selection decision with the stakeholders. Note that in this film evaluation problem, there are in fact two aggregation levels: aggregation of criteria scores for a given referee to get an overall score, and aggregation of referees’ evaluations to get a selection decision. According to the committee requirement, only the former aggregation level has been formalized. The latter does not receive a formal treatment: it is solved in an implicit way by the use of a decision support interface making it possible for the committee to have an idea of the distribution of grades over referee. It provides also each referee with some feedbacks on his internal judgment consistency (Hahn, 2010), and with a compact representation of the value system of each other under the form of qualitative influence indexes attached to each subset of criteria. These decision support functionalities aim at facilitating discussion among referees by providing information for preference analysis in order to lead to a shared decision with some legitimizations (Amgoud and Prade, 2009 and Belton and Pictet, 1997). The choice of not formalizing this second aggregation level is motivated by the organizing committee requirement to respect the referees’ opinions by letting them the possibility to have their own different value system, but also by the difficulty of the associated information processing. Indeed, the aggregation of overall scores given by all the referees require that these scores are seen as commensurate, this requires substantial efforts in the construction procedure of these scores. Concerning the first aggregation level, it could be considered by conventional weighted mean of numerical scores. The problem is that these numerical scores are difficult to elicit directly due to the human inability to have a precise perception leading to the identification of definite scores. Moreover, once they are apparently clearly intelligible to each stakeholder, numerical scores do not always have a sound measurement theory based meaning (Roberts, 1979) in terms of significance covered by the numerical values; e.g. the differences or the ratios are considered to have a meaning though the referees’ preferences are purely ordinal. However, there are a few methods to obtain numerical scores from qualitative preferences in a sound way. The Macbeth method (Bana e Costa & Vansnick, 1997) converts preferences into a numerical scale in a way which is coherent with the measurement theory but it requires an intensity of preference from the referees. The Tomaso method (Marichal, Meyer, & Roubens, 2005) provides numerical information. However, the final score is on a scale which has no relation with the original score scale, which makes it difficult to draw explanations for the stakeholders. One adapted way to obtain a clear and common elicitation of preferences regarding criteria and overall scores is to consider verbal terms on an ordinal scale. Thus, without fictitious numerical conversions, the evaluations preserve the original concrete point of view of each referee who is completely free to behave as he or she feels. In fact, this raises problems for the aggregation of these ordinal evaluations. Indeed, they are a lot of methods in multi-criteria decision making able to deal with ordinal scores such as Electre-like methods (Figueira, Greco, & Ehrgott, 2005), but they produce a rank of the alternatives and not a score as final result. Rule-based approaches can also be considered (Greco et al., 2005 and Zhang and Chu, 2009). They rather map alternatives to some predefined ordered categories, which could be considered as scores, but they have to be predefined. Moreover, only a few studies have been done in the domain of aggregation functions on finite ordinal scales and associated explanation capabilities have not been developed. Nevertheless, a recent method proposed by Grabisch allows to deal with the above practical preference processing challenges (Grabisch, 2006). His proposition consists in finding all the mean aggregation operators representing the referee preferences in a finite ordinal scale score context. By definition, a mean operator (in the wide sense) satisfies: internality (i. e. the overall evaluation should not be beyond the range of scores), and non decreasingness (that ensures that an improvement of one score cannot decrease the overall score). Thus this method allows to deal with consistency conditions on referees’ score profiles. Moreover, it also provides some responses to the questions raised by a stakeholder in order to obtain a better mutual understanding and thus legitimization of the final decision which is an important issue for the organizing committee. In summary, the paper presents a decision-making support system for animated film selection by using Grabisch’s method for the representation and aggregation of referees’ preferences. Section 2 describes the film evaluation procedure, i.e. the considered criteria, the finite ordinal scale scoring, and the determination of a mean aggregation operator. In Section 3, explanation functionalities about the referees’ behaviour are proposed in order to highlight the criteria or pair of criteria which are the most influential for each of the referees. Section 4 describes the developed interactive decision software which facilitates a shared selection procedure. Experimental results issued from the 2007 selection are presented in Section 5 and analysed from the preference representation and processing point of view.

A new selection process has been developed to facilitate the selection procedure of the Annecy Film Festival. In addition to the overall evaluation, four main criteria have been selected for the referees’ evaluation of film quality: Scenario, Aesthetics, Animation Technics and Soundtrack. A finite ordinal evaluation scale has been chosen to meet referees’ preferences. The evaluations’ inconsistency has been identified as preference non decreasingness and tradeoff. In order to explain each referee selection behaviour, which is useful for the final decision debate, a mean aggregation operator based on Grabisch’s method has been determined. The referees’ behaviour is presented to the decision stakeholders by criteria influences and criteria interactions expressed on a discrete ordinal scale. To facilitate a shared selection decision, software and hardware tools have been implemented to present the information in a suitable graphical form. This information allows referees to compare their preference behaviour and thus, to understand the causes of their conflicts. It is also useful for other stakeholders of the selection decision. The developed tools have been successfully experimented (less time spent, consensual decision-making, accountability) for the 2007 Festival selection. In the considered approach, the same number of levels, namely five, has been used for all the criteria and three for all the referees’ overall evaluations. It would be interesting to consider different level numbers, which is possible with Grabisch’s method, as well as more specific type of preference behaviour leading to more specific aggregation operators. This more elaborated approach will lead to more detailed explanations in relation with different capabilities of different referees and thus will constitute an interesting perspective in preference representation on an ordinal discrete scale.