asg/mem-alloc: Use doubly linked-list and update checker

Add a pointer to prev in struct block_meta and move unmodifiable files
to utils/.
Add memory leak checks and style checks to the checker.
Add make check-fast option to skip checking for memory leaks.
Add instructions for debugging with the VSCode debugger.

Signed-off-by: Alex Apostolescu <[email protected]>

content/assignments/memory-allocator/README.md

@@ -15,15 +15,16 @@ The goal is to have a reliable library that accounts for explicit allocation, re
 
 The support code consists of three directories:
 
-- `allocator/` will contain your solution based on the `osmem.h` header file
+- `src/` will contain your solution
 - `tests/` contains the test suite and a Python script to verify your work
-- `utils/` contains an implementation for `printf()` function that does **not** use the heap
+- `utils/` contains `osmem.h` that describes your library interface, `block_meta.h` which contains details of `struct block_meta`, and an implementation for `printf()` function that does **NOT** use the heap
 
 The test suite consists of `.c` files that will be dynamically linked to your library, `libosmem.so`.
-You can find the sources in the `tests/src/` directory.
-The results of the previous run can be found in the `tests/out/` directory and the reference files are in the `tests/ref/` directory.
+You can find the sources in the `tests/snippets/` directory.
+The results of the previous will also be stored in `tests/snippets/` and the reference files are in the `tests/ref/` directory.
 
-The automated checking is performed using `checker.py` that runs each test and compares the syscalls made by the `os_*` functions with the reference file, providing a diff if the test failed.
+The automated checking is performed using `run-tests.py`.
+It runs each test and compares the syscalls made by the `os_*` functions with the reference file, providing a diff if the test failed.
 
 ## API
 
@@ -75,6 +76,7 @@ The automated checking is performed using `checker.py` that runs each test and c
 
    - Allocations that increase the heap size will only expand the last block if it is free.
    - You are allowed to use `sbrk()` instead of `brk()`, in view of the fact that [on Linux](https://man7.org/linux/man-pages/man2/brk.2.html#NOTES) `sbrk()` is implemented using the `brk()`.
+   - Do **NOT** use [`mremap()`](https://man7.org/linux/man-pages/man2/mremap.2.html)
    - You must check the error code returned by every syscall.
    You can use the `DIE()` macro for this.
 
@@ -97,25 +99,23 @@ All memory allocations should be aligned to **8 bytes** as required by 64 bit sy
 
 We will consider a **block** to be a continuous zone of memory, allocated and managed by our implementation.
 The structure `block_meta` will be used to manage the metadata of a block.
-Each allocated zone will comprise of a `block_meta` structure placed at the start, followed by data (payload).
-For all functions, the returned address will be that of the **payload** (not the `block_meta` structure).
+Each allocated zone will comprise of a `block_meta` structure placed at the start, followed by data (**payload**).
+For all functions, the returned address will be that of the **payload** (not of the `block_meta` structure).
 
 ```C
-typedef struct block_meta {
+struct block_meta {
 	size_t size;
 	int status;
+	struct block_meta *prev;
 	struct block_meta *next;
-} block_meta;
+};
 ```
 
-_Note_: Both the `struct block_meta` and the payload of a block should be aligned to **8 bytes**.
-
-_Note_: The checker uses the `size` from `struct block_meta`.
-Use `size` to store the **aligned size of the payload**.
+_Note_: Both the `struct block_meta` and the **payload** of a block should be aligned to **8 bytes**.
 
 _Note_: Most compilers will automatically pad the structure, but you should still align it for portability.
 
-![memory-block](./assets/memory-block.svg)
+![memory-block](./img/memory-block.svg)
 
 #### Split Block
 
@@ -125,7 +125,7 @@ If we use one larger block the remaining size of that block will be wasted since
 
 To avoid this, a block should be truncated to the required size and the remaining bytes should be used to create a new free block.
 
-![Split Block](./assets/split-block.svg)
+![Split Block](./img/split-block.svg)
 
 The resulting free block should be reusable.
 The split will not be performed if the remaining size (after reserving space for `block_meta` structure and payload) is not big enough to fit another block (`block_meta` structure and at least **1 byte** of usable memory).
@@ -139,7 +139,7 @@ This is called [External Memory Fragmentation](https://www.tutorialspoint.com/di
 
 One technique to reduce external memory fragmentation is **block coalescing** which implies merging adjacent free blocks to form a contiguous chunk.
 
-![Coalesce Block Image](./assets/coalesce-blocks.svg)
+![Coalesce Block Image](./img/coalesce-blocks.svg)
 
 Coalescing will be used before searching for a block and in `os_realloc()` to expand the current block when possible.
 
@@ -164,35 +164,41 @@ This reduces the number of future `brk()` syscalls.
 For example, if we try to allocate 1000 bytes we should first allocate a block of 128 kilobytes and then split it.
 On future small allocations, we should proceed to split the preallocated chunk.
 
+_Note_: Heap preallocation happens only once.
+
 ## Building Memory Allocator
 
-To build `libosmem.so`, run `make` in the `allocator/` directory:
+To build `libosmem.so`, run `make` in the `src/` directory:
 
 ```console
-student@os:~/.../content/assignments/mem-alloc$ cd allocator/
-
-student@os:~/.../assignments/mem-alloc/allocator$ make
+student@os:~/.../mem-alloc$ cd src/
+student@os:~/.../mem-alloc/src$ make
+gcc -fPIC -Wall -Wextra -g -I../utils  -c -o osmem.o osmem.c
+gcc -fPIC -Wall -Wextra -g -I../utils  -c -o ../utils/printf.o ../utils/printf.c
+gcc -shared -o libosmem.so osmem.o helpers.o ../utils/printf.o
 ```
 
 ## Testing and Grading
 
-The testing is automated and performed with the `checker.py` script from the `tests/` directory.
+The testing is automated and performed with the `run-tests.py` script from the `tests/` directory.
 
-Before running `checker.py`, you first have to build `libosmem.so` in the `allocator/` directory and generate the test binaries in `tests/bin`.
+Before running `run-tests.py`, you first have to build `libosmem.so` in the `src/` directory and generate the test binaries in `tests/snippets`.
+You can do so using the all-in-one `Makefile` rule from `tests/`: `make check`.
 
 ```console
-student@os:~/.../content/assignments/mem-alloc$ cd tests/
-
-student@os:~/.../assignments/mem-alloc/tests$ make
-gcc -I../utils -fPIC -Wall -Wextra -g -o bin/test-all src/test-all.c -L../allocator -losmem
-gcc -I../utils -fPIC -Wall -Wextra -g -o bin/test-calloc-arrays src/test-calloc-arrays.c -L../allocator -losmem
-gcc -I../utils -fPIC -Wall -Wextra -g -o bin/test-calloc-block-reuse src/test-calloc-block-reuse.c -L../allocator -losmem
-gcc -I../utils -fPIC -Wall -Wextra -g -o bin/test-calloc-coalesce-big src/test-calloc-coalesce-big.c -L../allocator -losmem
-gcc -I../utils -fPIC -Wall -Wextra -g -o bin/test-calloc-coalesce src/test-calloc-coalesce.c -L../allocator -losmem
-gcc -I../utils -fPIC -Wall -Wextra -g -o bin/test-calloc-expand-block src/test-calloc-expand-block.c -L../allocator -losmem
+student@os:~/.../mem-alloc$ cd tests/
+student@os:~/.../mem-alloc/tests$ make check
+gcc -fPIC -Wall -Wextra -g -I../utils  -c -o osmem.o osmem.c
+gcc -fPIC -Wall -Wextra -g -I../utils  -c -o helpers.o helpers.c
+gcc -fPIC -Wall -Wextra -g -I../utils  -c -o ../utils/printf.o ../utils/printf.c
+[...]
+gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-all snippets/test-all.c -L../src -losmem
+gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-arrays snippets/test-calloc-arrays.c -L../src -losmem
+gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-block-reuse snippets/test-calloc-block-reuse.c -L../src -losmem
+gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-coalesce-big snippets/test-calloc-coalesce-big.c -L../src -losmem
+gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-coalesce snippets/test-calloc-coalesce.c -L../src -losmem
+gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-expand-block snippets/test-calloc-expand-block.c -L../src -losmem
 [...]
-
-student@os:~/.../assignments/mem-alloc/tests$ python checker.py
 test-malloc-no-preallocate       ........................ passed ...   2
 test-malloc-preallocate          ........................ passed ...   3
 test-malloc-arrays               ........................ passed ...   5
@@ -234,58 +240,46 @@ test-realloc-coalesce            ........................ passed ...   3
 test-realloc-coalesce-big        ........................ passed ...   1
 test-all                         ........................ passed ...   5
 
-Grade                            .................................. 9.00
+Total:                                                            90/100
 ```
 
+**NOTE:** By default, `run-test.py` checks for memory leaks, which can be time-consuming.
+To speed up testing, use the `-d` flag or `make check-fast` to skip memory leak checks, but remember to run `make check` before submitting your assignment to ensure it meets all criteria.
+
 ### Debugging
 
-`checker.py` uses `ltrace` to capture all the libcalls and syscalls performed.
+`run-tests.py` uses `ltrace` to capture all the libcalls and syscalls performed.
 
 The output of `ltrace` is formatted to show only top level library calls and nested system calls.
 For consistency, the heap start and addresses returned by `mmap()` are replaced with labels.
 Every other address is displayed as `<label> + offset`, where the label is the closest mapped address.
 
-To run a single test use `checker.py <test>`, where `<test>` is the name of the test without path or extension.
-Append `-v` to see the diff between `out/<test>.out` and `ref/<test>.ref`.
+`run-tests.py` supports three modes:
+
+- verbose (`-v`), prints the output of the test
+- diff (`-d`), prints the diff between the output and the ref
+- memcheck (`-m`), prints the diff between the output and the ref and announces memory leaks
+
+If you want to run a single test, you give its name or its path as arguments to `run-tests.py`:
 
 ```console
-student@os:~/.../assignments/mem-alloc/tests$ python checker.py test-malloc-split-one-block -v
-test-malloc-split-one-block      ........................ failed ...   0
---- out/test-malloc-split-one-block.out
-+++ ref/test-malloc-split-one-block.ref
-@@ -1,4 +1,23 @@
--os_malloc (['131040'])                                                                    = 0
--os_free (['0'])                                                                           = <void>
--DBG: os_malloc returned NULL on valid size
--+++ exited (status 6) +++
-+os_malloc (['131040'])                                                                    = HeapStart + 0x18
-+  brk (['0'])                                                                             = HeapStart + 0x0
-+  brk (['HeapStart + 0x20000'])                                                           = HeapStart + 0x20000
-+os_free (['HeapStart + 0x18'])                                                            = <void>
-+os_malloc (['65536'])                                                                     = HeapStart + 0x18
-+os_malloc (['32768'])                                                                     = HeapStart + 0x10030
-+os_malloc (['16384'])                                                                     = HeapStart + 0x18048
-+os_malloc (['8192'])                                                                      = HeapStart + 0x1c060
-+os_malloc (['4096'])                                                                      = HeapStart + 0x1e078
-+os_malloc (['2048'])                                                                      = HeapStart + 0x1f090
-+os_malloc (['1024'])                                                                      = HeapStart + 0x1f8a8
-+os_malloc (['512'])                                                                       = HeapStart + 0x1fcc0
-+os_malloc (['256'])                                                                       = HeapStart + 0x1fed8
-+os_free (['HeapStart + 0x18'])                                                            = <void>
-+os_free (['HeapStart + 0x10030'])                                                         = <void>
-+os_free (['HeapStart + 0x18048'])                                                         = <void>
-+os_free (['HeapStart + 0x1c060'])                                                         = <void>
-+os_free (['HeapStart + 0x1e078'])                                                         = <void>
-+os_free (['HeapStart + 0x1f090'])                                                         = <void>
-+os_free (['HeapStart + 0x1f8a8'])                                                         = <void>
-+os_free (['HeapStart + 0x1fcc0'])                                                         = <void>
-+os_free (['HeapStart + 0x1fed8'])                                                         = <void>
-++++ exited (status 0) +++
-
-
-Grade                            .................................. 0.00
+student@os:~/.../mem-alloc/tests$ python run-tests.py test-all
+OR
+student@os:~/.../mem-alloc/tests$ python run-tests.py snippets/test-all
 ```
 
+### Debugging in VSCode
+
+If you are using [Visual Studio Code](https://code.visualstudio.com/), you can use the [`launch.json`](.vscode/launch.json) configurations to run tests.
+
+Setup the breakpoints in the source files or the tests and go to Run and Debug (`F5`).
+Select `Run test` script and press `F5`.
+This will enter a dialogue where you can choose which test to run.
+
+You can find more on this in the official documentation: [Debugging with VSCode](https://code.visualstudio.com/docs/editor/debugging).
+
+If VSCode complains about `MAP_ANON` argument for `mmap()` change `C_Cpp.default.cStandard` option to `gnu11`.
+
 ## Resources
 
 - ["Implementing malloc" slides by Michael Saelee](https://moss.cs.iit.edu/cs351/slides/slides-malloc.pdf)

content/assignments/memory-allocator/src/Makefile

@@ -1,10 +1,12 @@
+UTILS_PATH ?= ../utils
+
 CC = gcc
-CPPFLAGS = -I../utils
+CPPFLAGS = -I$(UTILS_PATH)
 CFLAGS = -fPIC -Wall -Wextra -g
 LDFLAGS = -shared
 
 # TODO: Add additional sources
-SRCS = osmem.c ../utils/printf.c
+SRCS = osmem.c $(UTILS_PATH)/printf.c
 OBJS = $(SRCS:.c=.o)
 TARGET = libosmem.so
 
@@ -17,7 +19,7 @@ $(TARGET): $(OBJS)
 
 pack: clean
 	-rm -f ../src.zip
-	zip -r ../src.zip *
+	-zip -r ../src.zip *
 
 clean:
 	-rm -f ../src.zip

content/assignments/memory-allocator/src/osmem.c

@@ -1,7 +1,6 @@
 // SPDX-License-Identifier: BSD-3-Clause
 
 #include "osmem.h"
-#include "helpers.h"
 
 void *os_malloc(size_t size)
 {

content/assignments/memory-allocator/tests/Makefile

@@ -1,41 +1,44 @@
-SRC_PATH ?= ../src
+export SRC_PATH ?= $(realpath ../src)
+export UTILS_PATH ?= $(realpath ../utils)
+
 CC = gcc
-CPPFLAGS = -I../utils -I $(SRC_PATH)
+CPPFLAGS = -I$(UTILS_PATH)
 CFLAGS = -fPIC -Wall -Wextra -g
 LDFLAGS = -L$(SRC_PATH)
 LDLIBS = -losmem
 
-SOURCEDIR = src
-BUILDDIR = bin
-SRCS = $(sort $(wildcard $(SOURCEDIR)/*.c))
-BINS = $(patsubst $(SOURCEDIR)/%.c, $(BUILDDIR)/%, $(SRCS))
+SNIPPETS_SRC = $(sort $(wildcard snippets/*.c))
+SNIPPETS = $(patsubst %.c,%,$(SNIPPETS_SRC))
 
-.PHONY: all clean src check lint
+.PHONY: all src snippets clean_src clean_snippets check lint
 
-all: src $(BUILDDIR) $(BINS)
+all: src snippets
 
-$(BUILDDIR):
-	mkdir -p $(BUILDDIR)
+src:
+	$(MAKE) -C $(SRC_PATH)
 
-$(BUILDDIR)/%: $(SOURCEDIR)/%.c
-	$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(LDLIBS)
+snippets: $(SNIPPETS)
 
-src:
-	make -C $(SRC_PATH)
+clean_snippets:
+	rm -rf $(SNIPPETS)
+
+clean_src:
+	$(MAKE) -C $(SRC_PATH) clean
 
 check:
-	make -C $(SRC_PATH) clean
-	make clean
-	make -i SRC_PATH=$(SRC_PATH)
-	SRC_PATH=$(SRC_PATH) python checker.py
+	$(MAKE) clean_src clean_snippets src snippets
+	python run_tests.py
+
+check-fast:
+	$(MAKE) clean_src clean_snippets src snippets
+	python run-tests.py -d
 
 lint:
 	-cd .. && checkpatch.pl -f src/*.c tests/src/*.c
 	-cd .. && checkpatch.pl -f checker/*.sh
 	-cd .. && cpplint --recursive src/ tests/ checker/
 	-cd .. && shellcheck checker/*.sh
-	-cd .. && pylint tests/*.py
+#	-cd .. && pylint tests/*.py
 
-clean:
-	-rm -f *~
-	-rm -f $(BINS)
+snippets/%: snippets/%.c
+	$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(LDLIBS)