GSoD summary report

My GSoD task was to improve the Doxygen-generated documentation of ScummVM in terms of its structure, completeness, language, style, coherence, and overall usability.

Completed tasks

This is my final GSoD deliverable:

https://doxygen.scummvm.org/modules.html

To be clear, I have not created all of this from scratch. Doxygen comments have existed in ScummVM header files since forever, as they are a standard way of documenting C and C++ code, so that it can be used and understood by other developers (which is of huge importance especially in an open-source project).

However, the existing doxygen documentation was to a large extent incomplete and random. There was practically no structure to it, which I pointed out in my initial pull request to the repo that was part of my GSoD application. You can compare the screenshot of the old structure in that PR to the currently hosted version.

The specific tasks completed during GSoD involved:

1. Setting up doxygen build configuration. In the course of discussions with developers, we decided to bring the doxygen build framework from a separate repository to the main one and change the configuration to a large extent.

2. Refreshing the style of the doxygen output. This was a low priority task. I prepared 4 different CSS alternatives, uploaded them to my github.io server and presented to developers. The most favored option was chosen by popular vote.

3. Adding doxygen groups to all header files from the main folders of the scummvm repository to create a structured documentation set. This was done at the same time as the actual review of the files (next point).

4. Thoroughly reviewing all doxygen comments in the most important header files from folders ‘common’, ‘image’, ‘engines’, ‘graphics’, and ‘audio’.

In the course of GSoD, 12 of my pull requests were merged into the repository, all of them properly reviewed by code owners. The diffs in these pull requests demonstrate what kind of additions/changes I was making in these files. A typical ‘review’ of doxygen documentation in a header file involved:

• Making sure the output would appear correctly in the structure.

• Making sure every single C++ element from the file had doxygen documentation.

• Checking and fixing doxygen syntax, or adding doxygen elements to improve the final output.

• Checking and fixing doxygen errors, such as broken references or incorrectly documented arguments.

• Typical language and style check.

After opening a PR, I would always generate an HTML documentation set for the specific set of files covered in the PR and upload it to my github.io, to provide reviewers with a preview of how my changes affect the resulting output.

I was also documenting my progress by creating weekly blog posts which the organization kindly hosted on their Planet platform so that anyone interested could learn exactly what I was working on.

Prioritization

After 2-3 weeks of the project, both me and mentors realized we could approach in a more efficient manner. I started off reviewing the files alphabetically, whereas the correct way would be to approach them from the highest to lowest priority. The other reason for this was that the reviews were taking longer than expected and it became obvious that reviewing all header files in the large ScummVM codebase was unrealistic.

We decided to create a Google Spreadsheet that contained all the header files which were in scope of GSoD, and a mentor assigned a priority to all of them. We decided to focus on high priority files first and then, if possible, move to lower priorities.

All high priority files have been reviewed and a number of the lower priority files underwent a language and style review.

Current state of the project

At this point, the most essential and most frequently used header files in the project have complete and informative documentation. This means that open-source developers trying to work with ScummVM code can now rely on documentation much more than before. Whenever they are in doubt how to use a certain method or structure in their code implementation, the new doxygen documentation should help them a lot.

The documentation set is well-structured, which means it is now possible to find information by just navigating the menu on the left, rather than trying to search and hoping for the best.

All this is particularly critical when it comes to attracting new contributors to the project as it lowers the initial level of difficulty when working with the ScummVM codebase.

Follow-up work

The final deliverable is not perfect and still requires work. Firstly, only high-priority header files ended up in the scope of GSoD, plus a small number of lower priority ones. This means that there are still some blank spaces in the documentation that need to be filled. There are some files that lack doxygen almost completely, and a close cooperation of a technical writer and developers will be required to fix these cases.

Secondly, I will be creating a doxygen style guide so that developers are aware of the rules for using doxygen. Most importantly, that everything should be documented. This is essential in an open-source project where developers sometimes come and go. If they leave their piece of code undocumented, it might become a problem. I already have a full set of rules in mind but need to discuss with the main stakeholders of the project and then include it in documentation.

Another potential improvement is adding more content to the main landing page of this documentation. It is currently almost blank while it could serve as a source of useful information, like a description of the high-level structure of the codebase that would help new developers navigate the source code.

Challenges

The biggest challenge I faced was that I was essentially working on the same set of files that all other developers. My deliverable is not created from a separate set of documentation source files that nobody else was interested in apart from me. It is generated from ‘living’ code. So whenever I was changing something in a file on my fork, it was always highly probable that some developer was making changes there too, and we would end up merging at roughly the same time. This is the case especially for very active projects, like ScummVM.

As a consequence, while the review of my PR was ongoing, some merge conflicts would be appearing and some of them could be quite nasty. Because of one such conflict, I had to completely revert my review of one big file that took me several hours to complete.

Summary

Overall, I see this project as a big step towards having really valuable developer documentation in ScummVM. There are definitely some further improvements that can be made (and I will be working on them), but compared to the state of this documentation from three months ago, the progress has been considerable.

GSoD week 10

 Hi!

GSoD timeline is slowly coming to an end and I am slowly approaching the end of the list of high priority files defined for this project. As always, you can track my progress here:

https://docs.google.com/spreadsheets/d/1eslXDVzNvqnoQH1teHpNl_-ZdQnRgYve98lwCeTodyg/edit#gid=0

As for the status of the PRs, we got #2627 merged, which is great because it was massive amount of content. We even managed to avoid conflicts. Another big one still open and awaiting review is #2633.

As you can see in the spreadsheet, there a couple files left for me to review. Most of the stuff from the 'graphics' folder is done locally but I am still running some last checks on it. The PR (#2645) is already open with the first two files.I got some comments there from @criezy and addressed them. The rest of the graphics files will be added tomorrow.

Then, the only remaining task will be the four last headers - one from 'image' and three from 'audio'. Busy weekend ahead of me :)

GSoD week 9

 Hi!

Week 9 has been quite productive and I'm happy with the results, even though I haven't met my target of completing doxygen review on all high priority files.

The remaining files from the common folder are done and in PR#2627. They have been reviewed by @criezy (as always, huge thanks for the swift review) and I'm hoping we will get that merged soon.

I have also finished my work on headers from the engines folder and they can now be found in PR#2633. Four of these files have been categorized as high priority and to the remaining ones I have added a doxygen group definition so that they appear in the right place in the structure.

The engines headers gave me a bit of a headache and I had to spend much more time on them than initially planned. First challenge was to grasp all the concepts of engine, meta engine, Advanced Detector and how they relate to one another. I also had to make sure the references to various methods and structures that are declared in other files are resolved properly. Without any exaggeration, I can tell you that when reviewing a 700-line long file, I probably rebuild it about 50 times as I crawl through it, to make sure everything looks correctly. Fun!

I am now working on five files from 'graphics', they will be in PR very soon. Then one file from 'image' and three files from 'audio' and the high priority ones will be finally done.

This post is coming late (as usual), the next one will be here soon.

GSoD week 8

 Hi!

I'm posting a bit later than planned but I wanted to make sure one of the crucial PRs goes in before.

I've made some good progress this week and I am hugely thankful to @criezy for quickly jumping onto my reviews and leaving tons of great comments, some of which allowed me to understand the code better.

The biggest achievement of the week is that PR#2612 went in with some serious doxygen improvements in two big files: stream.h and system.h. Well over 2,5k lines of code overall. One significant change I made in system.h was the introduction of doxygen groups into the file (discussed in discord), so that now the documentation has a nice structure that is also reflected in the navigation pane on the left. I'm hoping this will improve the usability by a lot. Previously everything was presented on one page and because this is the largest header file in the scummvm codebase (or at least I believe so), this made the doc look a bit messy.

PR#2535 went in as well after a series of reviews.

Right now, I am preparing to open a PR for the last five high-priority files in the common folder (almost done locally). Then, I am moving on to engines, graphics, image, and audio. In each of these folders, we have identified a few header files as high priority.

For once I am happy with my progress this week and I'm truly hoping that in the next post I will be able to report that I am moving on to medium-priority files. Some of these have actually already been reviewed before we came out with the whole prioritization idea.

GSoD week 7

Hi!

I just realized that I completely forgot to write a blog post for last week so this one will be short and another one is coming in a few days.

I'm struggling quite hard right now to find time for the project and constantly feel that I am behind on the timelime. Unfortunately, my availability is hugely affected by the epidemic (the situation in Poland is really bad right now). Being locked up at home with two bored, attention-craving kids doesn't leave me much time for my regular job, let alone for GSoD.

The good news is that I got some great reviews on PRs #2535 and #2499. I have addressed all comments on the first one and it is now pending approval, while the second one is already merged.

One of these PRs contained tons of changes in the str.h file but in the end I had to revert all of them. This was due to the fact that in the meantime this file was changed extensively on master and resolving the resulting merge conflict would be too time consuming and too error prone for it to make sense. It is unfortunate to see a couple hours of work go to waste like that but I got used to the thought that it might and will happen.

I am finalizing a lot of changes locally and will be opening PRs during the weekend. I already made arrangements to ensure that I have sufficient time for GSoD in the next few days so hopefully you will see some more activity from me on Github and Discord.

GSoD week 6

 Hi!

Just a quick update this week.

I'm still waiting for someone to have a look at PRs #2535 and #2499. Hopefully they can get it soon.

I have quite a bit of work done locally but I'm not in a hurry to open another PR. My plan is to open a PR around the weekend with 10 more high-priority files with added/edited doxygen comments. The files in question will be the ones from stream.h up to metaengine.h from this list.

I am still investing a significant chunk of my time into developing the knowledge of aspects of C++ used in ScummVM headers. Figuring out how streams work was the latest challenge :)

GSoD week 5

 Hi!

The last week has been very busy and I've spent a lot of time working on finalizing PRs #2535 and #2499. The diff is quite massive so good luck to the reviewers :)

I am slowly crawling through the high-priority header files, as defined in the spreadsheet. Some of them can be really time-consuming, I think I spent over two days on str.h. A huge chunk of this time is spent on reading, googling, and expanding my knowledge of C++: how to understand operator functions, how iterators work, streams, arrays, lists, strings...I'm surely learning a lot but it annoys that it takes so much time that is then not really reflected in the work I'm submitting.

The problem I am already starting to see is that reviewing my PRs will take quite a bit of time. Every doxygen comment that I am adding there needs to verified, simply because my understanding of the function/type/attribute might have been wrong. And as my PRs are getting verified, the header files that they are massively editing are being changed on master by other developers, which produces some nasty merge conflicts. I guess fighting merge conflicts might become one of the main challenges in the remaining part of the project.

I'm going back to my files, you can track my progress in the Google spreadsheet if you are interested. I usually put a green 'Reviewed' field next to the files that I have already gone through and included in a PR.

GSoD week 4

Hi!

I'm doing this blog post a bit late when week 5 has already started. I've had some busy, tough days recently due to sick family (autumn and kids generally don't go well together) and couldn't devote as much time as I would like for my GSoD project. I'm a bit behind on the timeline right now but not really worried as I'm sure I can catch up with everything.

Right now PR#2499 is open with some more reviewed header files from the common folder. This is something I would probably like to get merged as soon as possible.

I've recently had some very good discussions with Eugene about the scope and priority of my work. You can check out this spreadsheet that contains a list of most headers from which we build doxygen documentation. The scope has been set for 114 files, most of which (75) reside in the common folder. Three levels of priority have also been assigned to the files.

Due to all this, I will be changing my approach to how I am working with the files. My assumption so far was that I would be going folder-by-folder, reviewing files alphabetically. My project timeline was also built around this idea, assigning particular weeks to work in particular folders. However, as there is clear priority now set on each header file, I am going to thouroughly review high-priority files first (there are 33 of those), then move to medium priority, then to low. This approach will assure that my GSoD project brings as much value as possible to ScummVM.

Stay tuned for PRs with some high-priority files in them!

GSoD week 3

 Hi!

Work is ongoing and I am still mostly focusing on reviewing doxygen blocks in the header files of the 'common' folder.

In the meantime, some things have been merged:

- PR#2361 - documentation generated from the 'common' folder is now nicely structured and divided into groups. Grouping will ultimately be applied to all header files in the project.

- PR#2457 - everyone can now easily build the doxygen documentation using a make command or the provided .sh script (big thanks to Thierry for his help with the scripts)

- PR#2467 - doc edits in header files

- PR#2488 - doc edits in header files

The last two PRs are merged but if you look into them, you'll be able to find my summary of the missing documentation which we haven't yet worked on. There is a lot to add and I can't do it on my own so any kind of help from the development team will be greatly appreciated. Creating developer documentation is always a cooperation between an SME and a tech writer so without you guys my powers are limited :)

During this week I also had a very interesting checkpoint call with my mentor Eugene where we clarified certain aspects of my GSoD project. I won't bore you with details but the scope and priority of tasks is now much better defined and I'm sure it's going to positively affect my progress.

GSoD week 2

 Hi!

Looking at the timeline of my GSoD proect, this week was supposed to be about refreshing the doxygen skin. I worked a bit with the doxygen CSS to make the doc output more coherent with the currect ScummVM website, for example using similar colors. This didn't turn out to be a great idea and decreased the overall readability of documentation, so in the end, this change has been reverted. As agreed through a Discord poll, we are going back to a vanilla doxygen skin for now.

At the same time, I am still working on header files from the common folder. Let me explain what I mean when say I'm 'working' on them:

•  Make sure each header has a @defgroup element with (at least) a decent group decription.
•  Make sure the language is grammatically correct and coherent. I will write a blog post with a summary of the styleguide that I'm using. It's really just in my head right now and should ideally end up as part of this documentation in the future.
•  Correct doxygen errors, for example:
• Typos in doxygen commands.
• Discrepancies when describing parameters in functions (different parameter name in the doxygen block vs what is in the code).
• Fixing broken references.
• Escaping characters that create problems. For example, doxygen will try to interpret anything within sharp brackets '<>' as an HTML/XML element.
• Add @name commands to group some related structures.
• What I should be doing more - trying to figure out and add descriptions for some functions or missing parameters. This is the hardest part for a non-developer who isn't very familiar with the project yet.

Another PR with some more reviewed header files is open now.

Week 3 and 4 are officially focused on documenting the common code. I got myself a bit of a headstart but this is a massive chunk of work so keep your fingers crossed for me!

GSoD week 1

Hi!

As the first week of my GSoD project is coming to an end, I'd like to give you a recap of what I've managed to accomplish so far.

I've worked a bit with my original pull request which was part of my GSoD application. I removed some files from it to restrict the scope and right now it is purely focused on adding doxygen groups to files in the common folder. This will greatly improve the structure of the doxygen output.We are still making some decisions there regarding terminology but I'm hoping this can go in pretty soon. You can view the output of that PR here.

Another PR opened right now is for the doxygen build framework. The idea is to put the doxygen build files in the main project repository. This will make it super easy for anyone who commits code into the repository to run a doxygen build on their updated header files, in order to check if the doc output is fine. You won't even need to push the files to do this, the build just runs locally from CLI using a simple doxygen command. Ideally, this kind of check could even become part of the ScummVM CI in distant future, but right now there are too many doxygen errors to even think about it. From my point of view, this PR is not very urgent - I have my local setup which I am using when rebuilding the doc a over and over as I work on the header files. What we commit into the repository needs to properly reviewed and discussed.

As to the main part of my work - according to the timeline, the first two weeks are about setting up the doxygen build and giving it a proper look. This is ongoing in the PR mentioned above. As the discussions there happen, I have started my work on the header files from the common folder, checking the language, doxygen usage and structure, adding missing information etc. I've got the first PR opened for a couple of header files and some more changed locally which will be put in a PR very soon.

The idea right now is to crawl over all of the header files alphatebitally and opening PRs for sets of 5-10 changed files. I don't think it makes sense to wait until I have done my work on all files from the common folder and only then open a pull request. That is 80 files and I don't want a discussion on one of them to block merging of all the other ones. Also, the longer the files remain in a pull request, the higher the risk of some nasty merge conflicts.

In week 2, I will continue my work in the common folder, hopefully reaching system.h within this week. This file is more important than the other ones and will require extra thourough review, and possibly extra support from the community to make sure everything in there is described in the proper way.

GSoD 2020 kick-off

Hi ScummVM community! 

I'm more than happy to announce that I've been chosen to participate in Google Season of Docs 2020 to work on ScummVM documentation. From now on, you will see me hanging around Github opening PRs as b-gent.

The aim of my project is to improve doxygen-based ScummVM API documentation that can be currently found here. I will be covering the details of my work in the upcoming posts in this blog. The plan still needs to be refined during the community bonding stage. For now, I can refer you to my project description which is now publicly available on the GSoD website.

I've done quite a bit of tech writing in my life so far, with 11 years of experience in various companies, big and small. I've created documentation for cloud applications, data warehouses, fintech solutions, various SDKs, and embedded devices. I've been good friends with markdown, DITA, RST, and (not surprisingly) doxygen. I've also been a teacher of technical documentation courses at some point, aimed at popularizing the profession of a tech writer in my country.

I am truly hoping to put all this experience to good use when working on this project.

On a more personal note, I live in Kraków, Poland. I've got two small kids who are pretty amazing, especially at driving me crazy every day. I like boxing and jumping rope. It really helps to vent all the stress related to work and parenting.

Looking forward to working with you!