Tools: Sketching

Sketching is a natural activity that people engage in while thinking and communicating, especially when spatial content is involved. Sketching is heavily used in STEM disciplines, such as engineering and geoscience. This working group is creating a sketch understanding system, called CogSketch, which is both a new research instrument for cognitive science research and a potential platform for sketch-based educational software. As a research instrument, we are using CogSketch to model spatial reasoning and learning, extending it using new results from SILC, and contributing to that science base in turn (e.g., a computational model of mental rotation). We are also exploring the use of CogSketch to gather and analyze data in behavioral experiments. Since the software can keep track of properties such as the timing of what is drawn, it can provide information that formerly would require video analysis to glean. Moreover, as SILC research progresses and its cognitive simulations become both broader and higher fidelity, we hope to be able to use it for scoring data, such as making similarity judgments. In education, we are exploring two paths to support STEM education. The worksheet model is designed to support students doing sketch assignments in class work. We are designing this to be as simple as possible, and as broad as possible, to support a variety of disciplines. To complement this effort, we are going in depth into an extremely difficult area, engineering design education. By exploring both in parallel, we maximize the likelihood of creating a truly versatile, useful platform for sketch-based intelligent educational software. Our vision is that, in ten years, sketch-based educational software could be as widely available to students as graphing calculators are today. Our hypothesis is that incorporating sketch understanding into educational software could provide revolutionary benefits for spatial education.

The aims of the Sketching Working Group are:

1. To model spatial reasoning and learning with CogSketch.
2. To explore the use of CogSketch in gathering and analyzing data in behavioral experiments.
3. To develop a worksheet model for CogSketch that can enhance STEM education.
4. To explore the use of CogSketch to support engineering design education.

Point of Contact:
Kenneth Forbus  

Downloads

• ♦Download CogSketch
• ♦Slides for CogSketch Tutorial

Publications from SILC

• ♦Forbus, K., Usher, J., Lovett, A., Lockwood, K. & Wetzel, J. (2008). CogSketch: Open-domain sketch understanding for cognitive science research and for education. In the Proceedings of the Fifth Eurographics Workshop on Sketch-Based Interfaces and Modeling. Annecy, France.
• ♦Jee, B. D., Gentner, D., Forbus, K., Sageman, B. & Uttal, D. H. (2009). Drawing on Experience: Use of sketching to evaluate knowledge of spatial scientific concepts. In N. A. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp 2499-2504). Amsterdam, The Netherlands.
• ♦Lockwood, K., Lovett, A., Forbus, K., Dehghani, M. and Usher, J. (2008). A Theory of Depiction for Sketches of Physical Systems. In the Proceedings of QR 2008.
• ♦Lockwood, K., Lovett, A., Forbus, K., Dehghani, M. and Usher, J. (2008). Automatic Interpretation of Depiction Conventions in Sketched Diagrams. In the Proceedings of the Eurographics Workshop on Sketch-Based Interfaces and Modeling.
• ♦Lockwood, K., Lovett, A. and Forbus, K. (2008). Automatic Classification of Containment and Support Spatial Relations in English and Dutch. In the Proceedings of Spatial Cognition.
• ♦Lovett, A. & Forbus, K. (2010). Shape is like space: Modeling shape representation as a set of qualitative spatial relations. AAAI Spring Symposium on Cognitive Shape Processing.
• ♦Lovett, A., Forbus, K., and Usher, J. (2010) A structure-mapping model of Raven's Progressive Matrices. Proceedings of CogSci-10.
• ♦Lovett, A. & Forbus, K. (2009). Computing human-like qualitative topological relations via visual routines. In Proceedings of the 23rd International Qualitative Reasoning Workshop. Ljubljana, Slovenia.
• ♦Lovett, A. & Forbus, K. (2009). Using a visual routine to model the computational of positional relations. In Proceedings of the 31st Annual Conference of the Cognitive Science Society. Amsterdam, The Netherlands.
• ♦Lovett, A., Gentner, D., Forbus, K. & Sagi, E. (2009). Using analogical mapping to simulate time-course phenomena in perceptual similarity. Cognitive Systems Research, 10, 216-228.
• ♦Lovett, A., Sagi, E., Gentner, D. & Forbus, K. (2009). Modeling perceptual similarity as analogy resolves the paradox of difference detection. In Proceedings of the 2nd International Analogy Conference. Sofia, Bulgaria.
• ♦Lovett, A., Tomai, E., Forbus, K. & Usher, J. (2009). Solving geometric analogy problems through two-stage analogical mapping. Cognitive Science, 33(7), 1192-1231.
• ♦Lovett, A., Dehghani, M. & Forbus, K. (2008). Building and comparing qualitative descriptions of three-dimensional design sketches. In the Proceedings of the 22nd International Qualitative Reasoning Workshop. Boulder, CO.
• ♦Lovett, A., Lockwood, K. & Forbus, K. (2008). A computational model of the visual oddity task. In the Proceedings of the 30th Annual Conference of the Cognitive Science Society. Washington, D.C.
• ♦Lovett, A., Lockwood, K. & Forbus, K. (2008). Modeling cross-cultural performance on the visual oddity task. In the Proceedings of Spatial Cognition 2008. Freiburg, Germany.
• ♦Lovett, A. Forbus, K. & Usher, J. (2007). Analogy with qualitative spatial representations can simulate solving Raven's Progressive Matrices. Proceedings of the of the 29th Annual Conference of the Cognitive Science Society. Nashville, TN.
• ♦Lovett, A., Forbus, K. & Usher, J. (2007). Using qualitative representations and analogical mapping to solve problems from a spatial intelligence test. Proceedings of the 21st International Qualitative Reasoning Workshop. Aberystwyth, U.K.
• ♦Lovett, A., Lockwood, K., Dehghani, M. & Forbus, K. (2007). Modeling human-like rates of learning via analogical generalization. Proceedings of Analogies: Integrating Multiple Cognitive Abilities. Nashville, Tennessee.
• ♦Lovett, A., Sagi, E. & Gentner, D. (2007). Analogy as a mechanism for comparison. Proceedings of Analogies: Integrating Multiple Cognitive Abilities. Nashville, Tennessee.
• ♦Wetzel, J. and Forbus, K. (2009). Automated Critique of Sketched Designs in Engineering. In Proceedings of the 23rd International Workshop on Qualitative Reasoning. Ljubljana, Slovenia.
• ♦Wetzel, J. and Forbus, K. (2009). Automated Critique of Sketched Mechanisms. In Proceedings of the 21st Innovative Applications of Artificial Intelligence Conference. Pasadena, California.
• ♦Wetzel, J. and Forbus, K. (2008). Integrating Open-Domain Sketch Understanding with Qualitative Two-Dimensional Rigid-Body Mechanics. In the Proceedings of the 22nd International Workshop on Qualitative Reasoning. Boulder, CO.
• ♦Yin, P., Forbus, K., Usher, J., Sageman, B., Jee, B. (2010). Sketch Worksheets: A Sketch-based Educational Software System. Proceedings of the 22nd Annual conference on Innovative Applications of Artificial Intelligence.

Additional References

• ♦Alvarado, C., & Davis, R. (2001). Resolving ambiguities to create a natural sketch based interface. Proceedings of IJCAI-2001, August 2001.
• ♦Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., et al. (1997). QuickSet: Multimodal interaction for distributed applications. Proceedings of the Fifth Annual International Multimodal Conference (Multimedia '97), (pp. 31-40), Seattle, WA. ACM Press.
• ♦Falkenhainer, B., Forbus, K.D., & Gentner, D. (1989). The structure-mapping engine: Algorithm and examples. Artificial Intelligence, 41, 1-63.
• ♦Feist, M.I., & Genter, D. (2001). An influence of spatial language on recognition memory for spatial scenes. In J.D. Moore & K. Stenning (Eds.), Proceedings of the 23rd Annual Conference of the Cognitive Science Society (pp. 279-284).
• ♦Forbus, K. (1995). Qualitative spatial reasoning: Framework and frontiers. In J. Glasgow, N. Narayanan, and B. Chandrasekaran, Diagrammatic reasoning: Cognitive and computational perspectives, AAAI Press.
• ♦Forbus, K., Ferguson, R., & Usher, J. (2001). Towards a computational model of sketching. IUI'01, January 14-17, 2001, Santa Fe, New Mexico.
• ♦Forbus, K.D., Gentner, D., & Law, K. (1994). MAC/FAC: A model of similarity-based retrieval. Cognitive Science, 19, 141-205. (Abridged version to be reprinted in T. Polk & C.M. Seifert (Eds.), Cognitive Modeling. Boston: MIT Press.)
• ♦Forbus, K., Nielsen, P. and Faltings, B. (1991). Qualitative Spatial Reasoning: The CLOCK Project. Artificial Intelligence, 51 (1-3).
• ♦Forbus, K., & Usher, J. (2002). Sketching for knowledge capture: A progress report. Proceedings of IUI'02, San Francisco, California, January 13-16, 2002.
• ♦Klenk, M., Forbus, K., Tomai, E., Kim,H., and Kyckelhahn, B. 2005. Solving Everyday Physical Reasoning Problems by Analogy using Sketches. Proceedings of 20th National Conference on Artificial Intelligence (AAAI-05), Pittsburgh, PA.
• ♦Kuehne, S., Forbus, K., Gentner, D., & Quinn, B. (August 2000). SEQL: Category learning as progressive abstraction using structure mapping. Proceedings of CogSci-2000. Philadelphia, PA.
• ♦Kuipers, B. (2000). The spatial semantic hierarchy. Artificial Intelligence, 119, 191-233.
• ♦Lockwood, K., Forbus, K., & Usher, J. (2005). SpaceCase: A model of spatial preposition use. In Proceedings of the 27th Annual Conference of the Cognitive Science Society. Stressa, Italy.
• ♦Lockwood, K., Forbus, K., Halstead, D. & Usher, J. (2006). Automatic Categorization of Spatial Prepositions. Proceedings of the 28th Annual Conference of the Cognitive Science Society. Vancouver, Canada.
• ♦Regier, T. (1996). The human semantic potential: Spatial language and constrained connectionism. Cambridge, MA: MIT Press.
• ♦Regier, T., & Carlson, L.A. (2001). Grounding spatial language in perception: An empirical and computational investigation. Journal of Experimental Psychology, 130, 273-298.
• ♦Tomai, E., Lovett, A., Forbus, K., & Usher, J. (2005). A Structure Mapping Model for Solving Geometric Analogy Problems. Proceedings of the 27th Annual Conference of the Cognitive Science Society, Stressa, Italy, 2190-2195.