Speaker: Gnana Bharthy 

Topic: Social Systems Modeling & Validation

Abstract:  A holy grail for social systems representation is a valid set of dynamic models that recreates the world virtually, and act as a test-bed, so that one can examine the effects that may arise from alternative courses of action. In an attempt to move towards this, we (speaker and his colleagues) have developed a multi-resolution framework based primarily on “thinking, feeling” cognitively detailed agents (but supplemented by other techniques such as systems dynamics and simple agents for scale up). It is a framework that synthesizes many best-of-breed models and best practice theories of human behavior and social interactions, and facilitates the codification of alternative theories. Such a virtual social system built primarily of cognitively detailed agents can provide multiple levels of correspondence (to its real counterpart), both at observable and abstract aggregated levels, but could also pose several challenges including large feature spaces, issues in information elicitation with database, experts and news feeds, counterfactuals, fragmented theoretical base, and limited funding for validation.

In this talk, we outline some of our past efforts in, and challenges faced during, evaluating models of social systems with cognitively detailed agents. We subscribe to the view that no model can faithfully represent reality, but detailed, descriptive models are useful in learning about the system and bringing about a qualitative jump in understanding of the system it attempts to model -- provided they are properly validated. Owing to richness in the model, we argue that the holistic approach and consequent continuous improvement are essential to evaluating complex social system models such as these.

Our own approach to model evaluation is to consider the entire life cycle and assess the validity under two broad dimensions of (1) internally focused validity/quality achieved through structural, methodological, and ontological evaluations; and (2) external validity consisting of micro validity, macro validity, and qualitative, causal and narrative validity. In this talk, we also elaborate on selected validation techniques that we have employed in the past. We recommend a triangulation of multiple validation techniques, including methodological soundness, qualitative validation techniques such as face validation by experts and narrative validation, and formal validation tests including correspondence testing. 

Based on: Bharathy, G. and Silverman, B. (2012. In-Press), “Holistically Evaluating Agent Based Social Systems Models: A Case Study”, Simulation Journal: Transactions of the Society for Modeling and Simulation International.