Introduction

Since built heritage is a legacy from the past, it is, by definition, meant to endure. Therefore, if we seek to analyze and better understand it, the dimension of time is essential. When studying documentary sources for a research project on a heritage site, researchers are not solely interested in the built environment’s condition at one specific point in time; they want to learn about the layout of the site at many points in time. It is the transformation process and, ultimately, the environment that led to such an evolution that are important. Given this keen interest in process, it seems only appropriate to include time, the fourth dimension, in digital models designed to illustrate the historian’s interpretation of the documentary sources associated with a heritage site.

In this way, a 4D model formalizes a process, a sequence of events. If we wish to generate digital models capable of adequately fulfilling this function, we must implement a system that is as flexible and robust as possible. The model must be easily modifiable so it can adapt to the potentially increased availability of documentary sources and the ongoing interpretation of such sources. The system must also be capable of handling large quantities of data so it can generate 4D models of heritage sites composed of several buildings that have evolved asynchronously.

With both of these requirements in mind, we set out to develop a 4D modelling protocol based on the synergy between a database and a 3D software (Autodesk Maya). The database compiles heterogeneous information (e.g., geometric, temporal and geographic data), while the geometric modelling engine generates the complex geometric shapes often found in old buildings. Our objective is to develop a new way of organizing and using information to improve the process by which knowledge about built heritage is transferred.

To test this protocol and stimulate debate about how to improve it, we conducted trials as part of the digital component of a vast research project on the city of Montreal, Canada. This research project examines the historic role of the former Canadian metropolis (now the largest city in Quebec) as a major crossroads for trade characterized by the free movement of goods, people and knowledge. We conducted a case study in co-operation with our partner, the Écomusée du fier monde. We began by modelling the evolution of a large-scale heritage site (an industrial complex) with the help of heterogeneous documentary sources, mainly plans, photos, maps and texts. We then followed this with the implementation of an interactive digital environment based on the synergy between a SQLite database and the 3D software linked by an algorithm written in Python.

In the following pages, we will begin by looking at existing research in the field of 4D modelling and by explaining why we do not make use of HBIM systems. Next, we will describe our proposed approach and discuss the modelling process. We will then describe our case study. Finally, we will share some preliminary findings and describe the avenues to be explored in the next phases of our research.

Existing research

In the field of built heritage, the function of a digital model can be to illustrate the interpretation of documentary sources associated with a given site. For over a decade, a number of researchers have been exploring the use of 4D modelling as a method for semantically enriching models (Hetherington and Scott 2004; Stefani 2010; Kersten et al. 2012; Rollier-Hanselmann et al. 2014). According to these researchers, if one wishes to create a useful depiction of a heritage site, it is essential to consider the time dimension of the built environment, in other words, the evolution of the site’s morphology.

When developing a 4D model, there are many advantages to using BIM (Building Information Modelling). Firstly, it can use 3D scanning data. With this method, the heritage site can be scanned and then modelled using the data acquired. A lot of the research that uses HBIM (Heritage Building Information Modelling) involves mapping parametric objects onto a point cloud (Fai et al. 2011; Oreni et al. 2014; Murphy 2012). Secondly, to document the current state of the built environment, users can label objects with attributes related to things like material degradation, structural deformations, etc. This information can then be used to model future phases of building restoration or site redevelopment (Ciribini, Mastrolembo, and Ventura 2015). Finally, this type of platform offers interoperability between the HBIM data and model and other tools like software for structural analysis, budget forecasting, etc. This interoperability enables various experts (engineers, architects, restorers, geologists, historians) to work together effectively on projects to restore damaged heritage sites.

However, despite the incredible potential of BIM, platforms like Autodesk Revit, Graphisoft ArchiCAD and Bentley Architecture do not seem appropriate tools for the work we are doing. This observation is based on three factors that we will comment on below: i) the function of the 4D model and the software tools that the data will be exported to, ii) the geometric limitations of the standard objects and the longevity of the parametric object libraries, and iii) the difficulty of mastering the tool.

Our research involves working with a group of historians. Our goal is to help these researchers disseminate their interpretation of a collection of documentary sources by developing a 4D model representing the evolution of the heritage site under study. We are not interested in depicting specific features of the building’s current state, just the process of evolution. The 4D models developed for this study are not designed to illustrate any gradual deterioration a building may have suffered (e.g., material degradation, site erosion, burial). Instead, our focus is on formalizing the consequences of human activity that have noticeably transformed the site at relatively specific points in time (construction, expansion, reassignment, demolition). Like Jones (2012), we are interested in the effects that human actions have on the built environment and are using groups of geometric shapes to represent the consequences of these actions.

In light of this approach (and the budgetary constraints associated with scanning large-scale sites), we opted not to scan our case study site. Therefore, the modelling process is not based on point cloud morphology, but exclusively on the study of the documentary sources available. In the models developed, we did not label objects with attributes related to material degradation and structural deformations. Our aim was not to organize future actions for the site in question, but instead to formalize its evolution, based on the analysis of the documentary sources. With regard to interoperability, we did not have to transmit data to any structural analysis or budgeting software. Since the goal of the modelling exercise was not only to illustrate the reasoning used by the historical researchers but also to disseminate this information, we did transfer data to a game engine.

The second factor concerns the inherent geometric complexity of older architectural forms, regardless of any degradation or deformation the building has undergone. BIM component libraries do not always appear to be suitable for the process of modelling a heritage built environment. Commenting on the process of modelling the Basilica of Santa Maria di Collemaggio, Oreni et al. expressed this as follows: “One of the main challenges for such a complex project was the limiting standardization offered by current BIM technology used to manage simple buildings and construction” (Oreni et al. 2014, 271). Indeed, BIM platforms are often developed to support the process of designing modern buildings using machined parts. However, old buildings were often designed by skilled craftsmen; this means a given type of architectural element could have numerous configurations. In light of this, when modelling an old building, it is essential to be able to i) create new types of components and ii) adapt them iteratively, depending on the configuration of the artefacts to be illustrated.

To address this issue, some researchers have developed parametric object libraries. For example, Murphy (2012) created a library of parametric objects based on architectural pattern books to document the classical architecture of Dublin between 1700 and 1830, while Stefani (2010) built parametric objects to model a French heritage site constructed in the year 6 BC (Tropaeum Alpium, located in the commune of La Turbie). This need to implement a library of components is underscored by Ciribini et al. in their description of the modelling phase in Autodesk Revit: “First of all, it was essential to carry out a customised BIM library, indeed the software used provides a wide range of BIM objects, but they were not suitable to achieve the research targets. The issues recorded during this phase reflect the complexity of parametrising historical elements in a software developed to model new buildings” (Ciribini, Mastrolembo, and Ventura 2015, 270).

Ciribini et al. also note how challenging it is to develop such a library: “This phase proved to be very time and resource consuming, indeed the uniqueness of each element of the building involved the need of achieving a very high degree of parametrisation” (Ciribini, Mastrolembo, and Ventura 2015, 270). Given the time and effort involved, the library created must therefore be exploitable in the medium term. However, in many cases, the procedures for generating parametric objects and the relationships between objects are coded in a language particular to the environment used and thus associated with a specific BIM platform or CAD software. For example, the parametric object library designed by Murphy (2012) was created as a Graphisoft ArchiCAD plug-in and coded in GDL (Geometric Description Language), while Stefani (2010) used MEL (Maya Embedded Language). In such cases, library permanence can be problematic because the system developed by the researcher relies on a specific software and thus depends directly on its “survival” in the decades to come.

Heritage research is sometimes carried out over relatively long periods of time because the research process is subject to the vagaries of funding. For example, for our case study, historians analyzed a collection of documents that was assembled in the 1990s. Should the research project ever be interrupted again for a long period of time, it is important that the files illustrating the interpretation of the documentary sources remain usable.

The third factor affecting our choice of technology concerns the make-up of our group of researchers. Rather than being experts from a variety of fields (engineering, architecture, geology, etc.), these individuals are all historians. However, it is important that team members somehow manage to master the digital tool used to create and update the 4D model. Although there are clearly multiple advantages to using HBIM, historians are not necessarily likely to use a BIM platform unassisted. According to Murphy (2012), in comparison to CAD software, it takes much longer to learn how to use a BIM platform. Given that the usability tests he ran involved participants with some technical training (e.g., groups of students from the Dublin Institute of Technology), we can assume that his conclusions apply a fortiori to individuals with training in the humanities, such as historians.

In short, while a BIM platform is an extremely complex, efficient system, it offers advantages we do not need for our work and presents certain drawbacks, notably with regard to the standard object libraries and the learning curve for users. Our proposed system is therefore coded in Python, a language used by a vast community of programmers, and based, in part, on a database. It also employs the geometric modelling engine of a 3D software program (Autodesk Maya) that can interpret this language. Python scripts can be used in tandem with 3D software such as Blender, Rhino, Maya, etc. We selected Autodesk Maya because, within the framework of this research project, the person who was in charge of developing the algorithm was already familiar with this software. We make use of Python in order to enhance system flexibility; few changes within the script would be necessary to modify the algorithm and replace Maya with another 3D software.

Logothetis et al. underscore the advantage of developing alternative solutions in the field of heritage building information modelling: “[…] in the field of open source no outstanding platforms for HBIM exist. […] In the area of open source platforms it seems that many more could be done to moderate the high prices in the existing commercial platforms.” (Logothetis, Delinasiou, and Stylianidis 2015, 182–183). Our objective is therefore to develop a compact, robust system focussed on the needs of research groups similar to ours, that is, a group of historians with a modest budget working to create 4D models of heritage sites using a collection of documentary sources. This system is designed to meet specific criteria: the data structure must facilitate updates (this ensures the model’s flexibility and encourages historians to master the tool), it must be possible to export data to game engines to disseminate the 4D model (and consequently disseminate the rationale of the historian interpreting the documentary sources) and the files must be archived in a format that remains exploitable in the medium term.

4D modelling process

Preliminary work on documentary sources

The work of the historian consisted of collating documentary sources, selecting and interpreting the relevant documents and establishing a chronology of the events that had marked the evolution of each building under study. The historian selected from among the following types of documents: plans and elevations, iconographic sources (photographs, engravings, and paintings), maps of the area (to put the site in context or formalize the district’s evolution) and textual sources (to provide information about a specific object, building or area).

We began by dividing the body of documentation into two parts: one contained all the documentation about the initial state of each building, while the other consisted of information about subsequent changes to the buildings. The analysis of the second group of documents helped identify the periods when the built environment underwent significant modifications. This made it possible to establish a timeline and divide it into sections. In some cases, there were gaps in the documentation for the periods selected. Consequently, it was necessary to identify which parts of the buildings were not adequately documented and create a tentative reconstruction for each one. For example, if a given façade did not appear in any of the iconographic documents, it was assigned a configuration similar to that of another façade on the building, which is a reasonable enough assumption (until new documentary sources become available).

Populating the databases

Working with the documentary sources associated with each of the buildings under study (plans, elevations, photos), we begin by dividing up the volume into various independent sections or components that, together, will make up the global model representing the building. This approach involves reconstructing the building’s morphology by assembling three distinct types of components: “Wall,” “Floor” and “Roof”. Each component may be associated with one (or more) sub-components: opening, cornice, pilaster, pediment, corner, colonnade, hole, stairs, etc. The database is populated by entering data regarding the type of architectural element, dimension, position, material etc. These values are to be provided, later on, as arguments for parametric objects.

The benefit of working with independent components is that the 4D model can be developed by gradually refining these components. For example, we start by defining the components that make up the walls (dimensions and position) and then define the sub-components (openings and other details) by adding successive entries to the database.

Moreover, within the body of documentation, some of the technical drawings did not indicate the scale. Ciribini, Mastrolembo, and Ventura (2015) talk about the limitations involved when using traditional documents to assign specific values to parameters that will be provided as an argument for parametric objects in a bank of architectural elements. To address this issue, in other words, when a technical drawing does not indicate the scale, or when there are only period photographs or artistic drawings of the building, we use proportions rather than precise dimensions for the modelling process. Elements modelled this way can be properly scaled at a later date when more precise documentary sources become available.

The data associated with the built environment are stored in a series of databases. One database contains the variables and data that apply to the site as a whole, while there is a set of databases for each of the main buildings on the site. For each of the main buildings, there are four databases (“initialization,” “addition,” “modification” and “disappearance”) dedicated to components and sub-components. Each of these four databases is composed of the same series of tables, and each table corresponds to either a component (“Wall,” “Floor,” “Roof”) or a sub-component (opening, cornice, pilaster, etc.).

When a project involves analyzing the evolution of a large-scale heritage site, historians often work with a very large body of documentation. To make information easier to locate, we distinguish, as we mentioned earlier, between data associated with the initial configuration of the components and data associated with subsequent changes to the building. The “initialization” database contains the data collected from the study of the first group of documents. This collection of documents is usually the largest because it documents the building in its entirety, while other collections are smaller, as these only relate to parts of the building that have been subjected to modifications or disappeared, or document only additions to the actual building. Since some knowledge of 3D modelling is required to divide the building into modules and populate the “initialization” database with values to be provided as an argument for parametric objects, these operations are usually performed by the modeller. (Note: The “modeller” is the person who knows how to use 3D modelling software and masters the programming language.)

The “addition,” “modification” and “disappearance” databases are used to record the data collected from the analysis of the second group of documents, that is, data associated with subsequent changes to the building. Populating these databases usually involves a smaller number of operations (compared to those in the “initialization” database), so it can be done by historians. For instance, to report a modification, one simply enters the period, code, level, name and attributes that have changed, while signaling a disappearance means entering the component’s year of disappearance, code, level and name.

It is vital that the databases be user-friendly so that both the modeler and historian can systematically populate the tables and make modifications as quickly as possible. Our key concern is to make sure that the system is sufficiently flexible for the modelling work to alternate with searches of documentary sources.

Generating and evaluating the model

Whenever the modeller or historian wants to check the current state of the model, he or she can move on to the generative phase. At this point, an algorithm reviews the database and exports, for each component, a list that includes all the data associated with this component and its various related sub-components (e.g., a “Wall” component and its openings, cornices and corners).

Buildings (as well as the land) are parametric objects and the databases are populated with values to be provided as arguments for these parametric objects. A Python file contains all the classes whose function is to generate the 3D geometry of architectural elements. Depending on the type of component, conditional procedures direct the flow of information towards the method that will generate the component’s geometry using Autodesk Maya’s geometric modelling engine. When the application is launched in Maya, it generates all the required components for all the applicable periods (see Figure 1).

Figure 1 

Data flow diagram.

Maya offers the option of developing a basic interface within the application itself. The sole purpose of the interface developed in Maya is to enable the modeller and historian to test the integrity of the various configurations of the model during the modelling process. Positioning the cursor at different spots along the timeline will cause different segments of the global model to be displayed. Components display quickly because all the 3D objects exist already and user activity never requires the generation of a new model. When implementing the 4D model, it is therefore possible to view and thus verify the models, without having to export them first.

Each component’s name is comprised of the concatenation of the character strings containing the following information: the period, the type of component, and the level. The nomenclature is a very important feature. Once the generating process is completed, a set of CSV files is automatically generated. These files contain the lists of names of the components to be displayed for each period studied. Since the model is now separate from the database, the elements in the character string naming the object will be used to filter the components, making it possible, for instance, to create a list that would only display components whose name contains the sequence “Level1.” This means users could ask to view only the ground floor of the set of buildings making up the site for each period studied (see Figure 2). The nomenclature of components thus enables the user to decide which components are displayed, leading to more efficient display management.

Figure 2 

The ground floor at two different time periods: 1947 (above) and 1995 (below). Usine Raymond industrial complex.

Case study

Our case study, conducted for the Écomusée du fier monde, involved the Usine Raymond, a former jam and pickles factory located in the Centre-South district of Montreal, Canada. This complex has undergone tremendous change over the years, in a district that is experiencing major development. Its first building was constructed in the early 20th century. Others were added, one by one, until the decline of manufacturing in the 1970s. The site continued to evolve, however, and in the 1990s it welcomed a new addition with a cultural mission: Theatre Usine C. The upcoming construction of a condominium development in the main building will soon add a new series of events to the timeline tracing the site’s evolution.

We implemented a 4D model based on the method described above. At the end of the modelling process, once the model’s various states are deemed acceptable, the next phase is to develop an interactive environment. This requires a visualization tool for implementing the various modes of interaction, such as links between 3D objects and images, automatic camera movements, sections of interactive text, etc. We used the Unity3D game engine to conduct our testing. This tool was selected for three reasons. Firstly, since the game engine is designed to implement interaction, it is possible to program the interface (in JavaScript) more concisely and systematically than in Maya. Secondly, it is not necessary to export the models to another format since Unity3D can read the Maya format. Finally, the system offers greater portability because it is possible to generate a standalone version that does not require the installation of Maya software.

We implemented a 4D digital environment that was made available to the museum’s visitors via an interactive touch screen. This digital environment provides access to both textual and iconographic sources; users can learn about the site location, the history of the factory complex, and the people and practices associated with these buildings. They also have access to the 4D model; this model enables them to move along a timeline and view the district’s evolution and the various industrial buildings constructed throughout the site’s history. Users have the option of viewing the built environment while interactively modifying the observer’s position. To explore the interior of the main buildings, they can create horizontal cross-sections of the model to view the various floors in the form of 3D plans. Furthermore, they have access to a list of iconographic sources such as engravings, period photographs and recent photographs. Once a document is selected, the camera angle shifts, bringing the observer’s view of the model in line with the viewing angle of the iconographic source displayed.

One of our key concerns is the user reaction to this type of digital environment. We want to assess how effective the 4D model is at arousing user interest, providing information and enhancing awareness. With this in mind, we conducted a series of usability tests. At the conclusion of this series of tests, the response to the digital environment was generally positive. The most popular feature seemed to be the links between the model, which enables users to find their bearings in the site, and the photographs, which provide a realistic impression of the location at various points in time (see Figure 3). Respondents also suggested several improvements for the interface, notably with regard to timeline interactions and the method for accessing photographs. These suggestions led us to make some modifications to the system. The newly improved digital environment has been the subject of a new series of user tests. This iterative process of testing and modifying the system was the subject of a recent article (Charbonneau, Robichaud, and Burgess 2015).

Figure 3 

Preliminary interface, developed within the framework of the research project, enabling the modeller and historian to test the integrity of the various configurations from within the Autodesk Maya environment (above) and preliminary interface in Unity3D (below).

Prospects

In the next phase of our research, we intend to enhance the system’s user experience by improving what Beacham (2012) calls the transparency of visualization. It is important that users be aware of which areas in the generated model are, in fact, tentative reconstructions. To date, the system’s databases contain no data on the documentary sources. The information stored in these databases is used solely to generate the geometry of 3D objects and the lists of elements to display. In the future, the databases will include information about the components’ level of certainty so the models displayed can use colour coding to convey the various degrees of certainty. The system will include a software dedicated specifically to managing documentary sources, like Zotero for example (www.zotero.org). We will link the software managing documentary sources to the visualization tool so users can make queries in order to access the iconographic sources affiliated with a given item, or generate a list of all the documentary sources arranged by ascending or descending level of certainty.

In our case study, the data used as arguments for parametric objects were recorded in the databases by the modeller. In an upcoming case study, we will study how the modeller and the historian work together. The modeller will populate the “initialization” database (recording parametric values for the initial state of the buildings) and develop the programming code describing the new parametric objects required to successfully carry out the modelling process. Following this phase, the historian, working with the analysis of the documentary sources, will record the data in the “addition,” “modification” and “disappearance” databases to account for any modifications made to the built environment. We will then be able to verify which aspects of interacting with a database are problematic for the historians and assess the need for developing an interface adapted to their needs.

Conclusion

In this research project, the implementation of 4D models is based on the synergy between Autodesk Maya software and the SQLite database. Maya is used to generate the digital models, while SQLite stores and sorts the information. Our objective was to implement a system that was as flexible and robust as possible for generating 4D models. The aim was not to reinvent the wheel, but to put forth an alternative 4D modelling solution that could be adapted to the needs of a group of historians who did not want to use BIM, for both technical and budgetary reasons. We wanted the model to be easily modifiable so it could adapt to the potentially increased availability of documentary sources and the ongoing interpretation of such sources. We also wanted the system to be capable of handling large quantities of data so it could generate 4D models of heritage sites composed of several buildings that have evolved asynchronously.

The system we implemented enabled us to control all the aspects of the modelling process so we could make any necessary changes. The model’s flexibility lies not only in its ability to effectively modify the morphology of architectural components, but in its ability to quickly regenerate multiple partial versions of the model.

In addition to the case studies presented in this article, our work has led us to model other heritage buildings located in Montreal (e.g., the Narcisse-Desmarteau store warehouse, the Montreal & Southern Counties Railway terminal, Viger Station, Windsor Station, and the St. Anne’s Market building housing the Parliament of United Canada). In each of these case studies, we had to model a set of architectural components and sub-components specific to the buildings under study. In this way, we have optimized, enriched and gradually refined our library of parametric objects documenting 19th- and 20th-century Montreal architecture. Since it is coded in Python, this library will be exploitable in the medium term; its permanence will not depend on the “survival” of proprietary software in the decades to come.

Further to our case study, we can confirm that the system implemented has an acceptable level of flexibility; it is relatively easy to enter data in the databases to reflect the modifications made to the built environment over the years. This flexibility is enhanced by the fact that numerical values may be specified at any time during the modelling process (e.g., after surveying the site or obtaining new documentary sources with the scale indicated). Moreover, we demonstrated that it is possible to import such a model into a game engine and implement a 4D digital environment effective at arousing user interest, providing information and enhancing awareness.