Sugar Live build no longer shows changes.

[chimosky]

Noticed this while I ran genpot for typing-turtle-activity, tried to see the value of args in cmd_genpot of bundlebuilder and I didn't get any output.

Also noticed with sugar#852 and sugar#851.

@quozl.

[quozl]

Sorry, I don't understand. What do you mean by shows live changes?

[ghost]

On Tue, Feb 18, 2020 at 12:56:40PM -0800, James Cameron wrote: Sorry, I don't understand. What do you mean by shows live changes?

I'm also uncertain. There are a lot of different ways to use sugar-live-build in which something can go wrong. Are you using the image James has built, dowloaded and booted in a virtual machine of some kind? If so, what virtual machine? Virtual Box, kvm? If kvm, how invoked? Is it booted with the image ephemerally, as if from a virtual CD/DVD drive? In this case, one expects changes not to persist between boots. Has the image been used to install to a virtual hard disk, in which case one expects changes to persist on that hard disk? Are you using sugar-live-build to create your own image? Is this about trying to add functionality to the image that results from that process? I so like to use script(1) (from the bsdutils package on Debian) for recording and communicating what happens in the terminal. I wonder if it would be useful here? https://manpages.debian.org/stretch/bsdutils/script.1.en.html

[chimosky]

Sorry, I don't understand. What do you mean by shows live changes?

Edited title.

What I meant was changes made to the toolkit or sugar itself doesn't persist.

Are you using the image James has built, dowloaded and booted in a virtual machine of some kind? If so, what virtual machine? Virtual Box, kvm? If kvm, how invoked?

The image James built had some issues with graphical installation and I built mine after someone fixed it, you can get it here.
It's booted in gnome-boxes.

Is it booted with the image ephemerally, as if from a virtual CD/DVD drive? In this case, one expects changes not to persist between boots.

No it isn't.

Has the image been used to install to a virtual hard disk, in which case one expects changes to persist on that hard disk?

Yes it has and yes the changes are expected to persist but since the port to python3 changes no longer persist.

Are you using sugar-live-build to create your own image? Is this about trying to add functionality to the image that results from that process?

No I'm not.

I so like to use script(1) (from the bsdutils package on Debian) for recording and communicating what happens in the terminal. I wonder if it would be useful here?

Yeah I know of script but it'll not be useful here.

How to reproduce

• Make a visible change to the toolkit or sugar - add a print statement -.
• Look at logs if you'll see the statement printed.

You can also test the PRs I mentioned above, I had a python2 - sugar live build - machine running before now but I lost that due to some issues. @quozl if you have any you can test the mentioned PRs above and also test same rebased on master, If you do have any of those images I'd like to have it. Thanks.

[quozl]

Tell me what you do after you make a change to the Toolkit or Sugar?

Have you verified the change is in the files in the operating system? i.e. in /usr/lib

Don't forget, sugarlabs/sugar#868 means you have to stop Sugar or an activity before looking at the log file.

[quozl]

I'll speculate some more. Perhaps you accidentally installed the toolkit or Sugar into a different directory. Please use strace to verify the files used.

e.g.

$ strace -e openat -o tmp python3 -c 'import sugar3'
$ grep sugar3 tmp
openat(AT_FDCWD, "/usr/lib/python3/dist-packages/sugar3/__pycache__/__init__.cpython-36.pyc", O_RDONLY|O_CLOEXEC) = 3

Above demonstrates on this system that the Toolkit is being used from /usr/lib

[chimosky]

Tell me what you do after you make a change to the Toolkit or Sugar?

I restart.

Have you verified the change is in the files in the operating system? i.e. in /usr/lib

I'll verify but normally that should happen after building.

[quozl]

I restart

What is it you restart? An activity, Sugar, or the operating system?

[chimosky]

I restart

What is it you restart? An activity, Sugar, or the operating system?

I restart sugar when changes are made to sugar or the toolkit.

[chimosky]

I'll speculate some more. Perhaps you accidentally installed the toolkit or Sugar into a different directory.

I'm sure that's not the case.

[quozl]

Okay, then I suggest you use the strace method to confirm which file is being imported, and check that the file being imported contains the change you made in git. As you've not been specific about the change you made, I can't give exact instructions for this.

[chimosky]

Output of strace method

openat(AT_FDCWD, "/usr/lib/python3.7/dist-packages/sugar3/__pycache__/__init__.cpython-37.pyc", O_RDONLY|O_CLOEXEC) = 3

Okay, then I suggest you use the strace method to confirm which file is being imported, and check that the file being imported contains the change you made in git. As you've not been specific about the change you made, I can't give exact instructions for this.

The strace method confirms the file being imported but the file doesn't contain the changes I made in git.
The change I made was to print out args in cmd_genpot of bundlebuilder.py.

I rebuilt sugar after the change and then restarted sugar.

[quozl]

How did you rebuild and reinstall Sugar?

[chimosky]

How did you rebuild and reinstall Sugar?

I ran autogen.sh with --with-python3 arg - for the toolkit -, then make and make install.

[quozl]

Thanks. You haven't run autogen.sh correctly. By omitting the --prefix /usr done by 0900-sugar.hook.chroot the result will be that make install will put the files in /usr/local, and since you already have files in /usr from a previous make install they will be used instead. You can prove that by looking for sugar3 in /usr/local.

If this isn't the case, please use script(1) to catch a log of how you rebuild and install Sugar, and put it in a temporary Gist. Perhaps if I'd seen that I would have noticed /usr/local or the different autogen.sh command line immediately.

[quozl]

Also note the surprising rsync -r needed because of how Debian policy handles the Python path.

[chimosky]

Thanks. You haven't run autogen.sh correctly. By omitting the --prefix /usr done by 0900-sugar.hook.chroot the result will be that make install will put the files in /usr/local, and since you already have files in /usr from a previous make install they will be used instead. You can prove that by looking for sugar3 in /usr/local.

Just learning that I'm supposed to include --prefix /usr as used by the toolkit and it was introduced after the port to python3, the repos don't state that autogen should be run with that either.
Checked /usr/local and sugar doesn't exist there.

If this isn't the case, please use script(1) to catch a log of how you rebuild and install Sugar, and put it in a temporary Gist. Perhaps if I'd seen that I would have noticed /usr/local or the different autogen.sh command line immediately.

Will do.

[chimosky]

Gist here.

[quozl]

Thanks.

No, --prefix was always available as part of autotools.

No, it wasn't introduced after our Port to Python 3.

No, the README.md of the components don't state that nor should they. That could be a disaster for anyone managing a normal system with packaged software.

Checked Gist. It has installed to /usr/lib/python3.7/site-packages, yet your import evidence shows /usr/lib/python3.7/dist-packages, so you will certainly need that rsync command to cover this Debian policy effect.

[chimosky]

Checked Gist. It has installed to /usr/lib/python3.7/site-packages, yet your import evidence shows /usr/lib/python3.7/dist-packages, so you will certainly need that rsync command to cover this Debian policy effect.

I'll run rsync but the image I built was from same repo so I'm wondering why.

[quozl]

I'll run rsync but the image I built was from same repo so I'm wondering why.

The problem has nothing to do with installing from a different image. It has to do with how you rebuilt and installed the toolkit on the installed system.

[chimosky]

The problem has nothing to do with installing from a different image. It has to do with how you rebuilt and installed the toolkit on the installed system.

I said so because the image built contained the rsync line and that should have been executed so I wouldn't have to execute it myself each time I make a change, just noticed I have to execute that before building and installing to see my changes.

[chimosky]

Changes now persist but I have to run the rsync command each time a change is made.

I didn't have to include --prefix.

[quozl]

The rsync line is in the sugar-live-build image builder, but not in the sugar-toolkit-gtk3 toolkit. That's why you have to execute it.

You do have to use --prefix if you have done a git clean and removed autotools configuration cache.

You're using the method that I was willing to publish. I chose that one because it was easiest to explain and support. There are other methods for changing and testing the toolkit if you need to explore them;

• placing a copy of the modified toolkit in a directory sugar3 inside an activity, causing the activity to import it instead of the system-wide toolkit; suitable for developing a fix to an activity,

• placing a copy of the modified toolkit in a directory sugar3 in the user directory, causing Sugar to import it instead of the system-wide toolkit; suitable for developing a fix to Sugar,

• as root placing /usr/lib/python3.7/dist-packages under change control using git, editing the files directly as root, and using the repository as a git remote for your development repository, to cherry-pick changes as you go; suitable for developing a fix to either Sugar or an activity,

• as root copying files using a script from your development repository through ssh into the VM; this is my favourite.

[quozl]

Summary: after a build, install, and restart, changes to toolkit were not seen. Cause was not following the same build and install sequence as was used when the image was built.

[chimosky]

The rsync line is in the sugar-live-build image builder, but not in the sugar-toolkit-gtk3 toolkit. That's why you have to execute it.

Isn't it supposed to be executed when the image was built?

[quozl]

The rsync line is in the sugar-live-build image builder, but not in the sugar-toolkit-gtk3 toolkit. That's why you have to execute it.

Isn't it supposed to be executed when the image was built?

Literal answer; yes, it is executed when a Sugar Live Build image is built. You can prove that happened by finding the sugar3 module installed in dist-packages. Unless it was executed, Sugar would not start.

I've no idea what is behind your question though. I thought it was self-evident from reading the code and using Sugar Live Build. You'll have to tell me if there is another aspect to your question.

[chimosky]

I've no idea what is behind your question though. I thought it was self-evident from reading the code and using Sugar Live Build. You'll have to tell me if there is another aspect to your question.

Yes it's self evident but it seems that line doesn't get executed as I have to run it myself.

[quozl]

Thanks. I guess I'm missing why you mention it. You have to run it yourself, in the same way that you have to edit the source files, run autogen, make, make install, git clean, and repeat for Python 2 if necessary. The same steps in slightly different form are in Setup a development environment - Native Sugar and in RPM spec files and Debian rules files. The copying or moving of files to dist-packages is a consequence of a Debian policy that aims to separate Python modules installed by Debian packages from Python modules installed from source directories. If you do find a better way to overcome this policy, make a pull request.

[quozl]

@free-libre-software, this issue may be of interest.

[Saumya-Mishra9129]

I tried as suggested in https://github.com/sugarlabs/sugar-live-build/blob/master/src/config/hooks/normal/0900-sugar.hook.chroot#L42 for building sugar after making changes still changes were not visible. This is what I did - install-sugar.log. And then I do a log out from sugar. Still no changes was visible, I couldn't figure out what I did wrong. @quozl @chimosky can you please look once.

[Saumya-Mishra9129]

https://github.com/sugarlabs/sugar-live-build/blob/master/src/config/hooks/normal/0900-sugar.hook.chroot#L42, command was of no interest for me as the jarabe folder was present in dist-packages directory already, and there were no jarabe folder in site-packages.

[radiidev]

Its difficult to judge from log if all steps were followed since mv by default does not give feedback when successfully processed. But 'ok' in log suggest that you did follow all the steps and mv operation was successful. Though there are no conditional && operator inside parentheses. So, my assumption is that it will output 'ok' even if some operation exited abnormally.

I followed additional step mentioned in mailing list i.e.:

rm -rf /usr/lib/python3.7/dist-packages/jarabe/

mv without --force flag will fail if dist-packages/jarabe/ is not empty. It's better to remove dist-packages/jarabe/ before moving site-packages/jarabe to it even with --force flag as pre-existing and new files might get mixed up.

https://github.com/sugarlabs/sugar-live-build/blob/master/src/config/hooks/normal/0900-sugar.hook.chroot#L42, The file was of no interest for me as the jarabe folder was present in dist-packages directory already, and there were no jarabe folder in site-packages.

site-packages/jarabe is likely empty because your mv operation was successful which renamed it to dist-packages/jarabe or maybe some make install failure aborting populating it? dist-packages/jarabe is from where sugar loads. That is why, we need to copy compiled files from site-packages to dist-packages.

Can you try following process:

#!/usr/bin/bash

cd /usr/src/sugar && (
./autogen.sh --prefix /usr \
&& make \
&& make install \
&& rm -rvf /usr/lib/python3.7/dist-packages/jarabe/ \
&& mv -v /usr/lib/python3.7/site-packages/jarabe /usr/lib/python3.7/dist-packages/
) > /usr/src/install-sugar.log 2>&1 \
&& echo OK

Save it in a file like update.sh and than run it as:

$ sudo bash update.sh

Also, maybe try rebooting instead of log out and log in as well.

[Saumya-Mishra9129]

Thanks @free-libre-software it worked for me now.

&& rm -rvf /usr/lib/python3.7/dist-packages/jarabe/ \

The cause why it wasn't working is this line. After adding it worked fine.

[quozl]

The set -e makes bash stop if any step returns non-zero, so tha's why && wasn't used here, but there's no reason why it could not be.

https://github.com/sugarlabs/sugar-live-build/blob/master/src/config/hooks/normal/0900-sugar.hook.chroot#L5

Also, the script is run on a system that doesn't have Sugar installed. Once Sugar is installed, the extra step is needed to remove the directory before moving the new one into place. Since both these extra steps are specific to Debian, they don't belong in Sugar itself.

Also, when you're not sure if your change is present, check if it is. Use diff or view the changed file in the installed path. There is no purpose in restarting or rebooting if the file is unchanged.

Lastly, logging out and logging in to Sugar is sufficient for testing changes to Sugar or Toolkit. Restarting or rebooting should not be necessary. An exception is when testing Datastore; the D-Bus process can be preserved from a previous session if you log in within a minute.

[Saumya-Mishra9129]

I recently tried to do some changes in sugar in jarabe folder , ran that script. Changes were available, I don't even need to do logout or reboot. But when I did a reboot later, something weird happened I don't know what and now I am stuck in a login loop. I am not able to do a login again. I have noticed this problem some days ago also.

[chimosky]

Your change must have caused it, if you can ssh into the machine you should be able to check shell.log for what the problem is, fix it and build again.

[radiidev]

I recently tried to do some changes in sugar in jarabe folder , ran that script. Changes were available, I don't even need to do logout or reboot. But when I did a reboot later, something weird happened I don't know what and now I am stuck in a login loop. I am not able to do a login again. I have noticed this problem some days ago also.

As @chimosky said. Changes made must be crashing sugar. Press Ctrl+Alt+F1 to switch from GUI to CLI (beware, you can likely switch your parent OS to CLI if using GNU/Linux. So, you need to send signal to virtual machine instead of literal keyboard press). Switching back to GUI for live build is Ctrl+Alt+F7. Once in CLI, enter username & password to login. Check shell.log for errors and fix it or reverse changes you made. For editing source (text) files, you can use nano CLI editor.

[Saumya-Mishra9129]

Your change must have caused it, if you can ssh into the machine you should be able to check shell.log for what the problem is, fix it and build again.

Get it solved by switching to virtual console and removing all changes.

[quozl]

A login loop usually means Sugar is failing to start, and you'll find an explanation briefly in the shell.log file. If you have automatic login enabled, you'll have to use SSH or a text virtual console to regain control of the VM. Keep good snapshots of your virtual machines. Always set up inbound SSH on your virtual machines.