[Feature request]: JASP should support more than 1000 columns

[JeffS1984]

JASP Version

0.18.0

Commit ID

No response

JASP Module

Unrelated

What analysis are you seeing the problem on?

No response

What OS are you seeing the problem on?

Windows 11

Bug Description

I have been using JASP for a couple years to teach undergrad stats. This is the first semester I've had student issues who are using Windows. I've had a few who are unable to get Jasp to upload a .csv data set. The error they receive refers to "alter tables". I have attempted to work with these students to get Jasp 18 totally deleted from their computers and download a slightly older version like 17.2. However, their computers keep saying they already have Jasp downloaded, even though we cannot find Jasp anywhere on there.

Expected Behaviour

The .csv file should pull up on Jasp

Steps to Reproduce

1. Download Jasp 18 on Windows
2. Attempt to upload a .csv file
3. I'm not sure if this occurs for all Windows users -- I've only heard of an issue from 2-3 students out of 85-90.
   ...

Log (if any)

No response

Final Checklist

• I have included a screenshot showcasing the issue, if possible.
• I have included a JASP file (zipped) or data file that causes the crash/bug, if applicable.
• I have accurately described the bug, and steps to reproduce it.

[boutinb]

Hi @JeffS1984 This is quite annoying, but to be able to help you, we need a reproduction path. Without it, we cannot reproduce this error, and it is quite hard to see what is going wrong.

[JeffS1984]

My student was able to get past the issue. However, I updated my JASP to version 18 and am now running into the same thing. I have a Mac -- Ventura 13.3.3.1 and Apple M1 chip.

I am attaching a picture of the error when trying to upload a .csv file that has worked just fine with previous versions of Jasp.

I also noticed that on a data set that Jasp would upload, it was not producing basic output for me. I did a screenshot of that as well.

Thanks.

[boutinb]

Hi @JeffS1984 How many columns have your datafile? The error seems to indicate that there are more than 1000 columns. We know about this limitation, and we want to add some setting to circumvent this problem, but first just to be sure: have you more than 1000 columns. If yes, would it be possible to split this datafile with less columns?

[ertugruluyar-github]

I got same issue when I tried to open PISA 2022 student data.
I am using Linux and I installed JASP via flatpak (flathub).

[boutinb]

Hi @ertugruluyar-github, does have this PISA 2022 student data more than 1000 columns?

[ertugruluyar-github]

Hi @ertugruluyar-github, does have this PISA 2022 student data more than 1000 columns?

Hi @boutinb
Yes, there is 1279 columns.
I tried with Jamovi 2.4.11 also but the progress bar not filling and it doesn't give any error message and I wait hours and hours but it didn't open.
I tried with GNU PSPP also and it did load the variables, but not the data.
So, because of all softwares that I tried didn't able to open the .sav file, I write some python code with chatGPT and filtered data with country id. And Jamovi be able to open file (I haven't filtered the columns.).
Some columns in the dataset not so needed, so I can write some code with chatGPT to filter only variables that ı specified. So that, JASP could open the dataset. But why 1000 columns limit is there?

To be clear, I want to open the file because I wanted the filter datates with country id then cut out the the dataset to have small .sav file to analyse it. I want to have a small .sav beacuse my laotop not so powerful and 1.4 GB PISA 2015 file opened with Jamovi in like 30-40 minutes.

To be noted, I wanna write my thesis with Hiyerarcihal Linear Modelling (Multilevel Modelling) and in JASP there is a tool for that but in this tool I didn't able to analyse ANOVA model (fully unconditional model). If it has this (Maybe I don't know to do this.), because it have the Open Science Framework (https://osf.io/) integration, I wanted to use it for my analysis even the Jamovi's multilevel anlysis tool a bit better for me.

[tomtomme]

@ertugruluyar-github @JeffS1984
just a status update. This is still valid with current 0.19 beta. It will not be forgotten.

[JorisGoosen]

Some columns in the dataset not so needed, so I can write some code with chatGPT to filter only variables that ı specified. So that, JASP could open the dataset. But why 1000 columns limit is there?

Because we switched to using sqlite as an internal store and they apparently have a hardcoded maximum amount of columns in a dataset. We only noticed this after all the code was already written. Because usually people dont use JASP for such datasets. More things like R etcetera.

Im going to assume that even if it would load there might be other bottlenecks in the application (when loading it in R or something, not sure though).

We are working on syntaxmode, which might obviate the need for the sqlite storage and thus might be able to handle this many columns.
Of course, supporting an arbitrary amount of columns is desirable, but given our aim at teaching and making statistics available for non-computer-expert-users we are focussing first on things like supporting data editing, speed and handling different kinds of data.
The >1000 columns use-case is certainly a real one but it is a bit more niche so it might be a bit of time before we get to this

[tomtomme]

@JeffS1984
Can you alter the title of this issue to reflect that it is now a feature request to support > 1000 columns? Thx

[JeffS1984]

Thanks for all the comments on here. I do teach undergraduates with Jasp, but I like to use large social surveys like the General Social Survey in the US, which is why this issue came up for me.

[tomtomme]

This issue is tracked via an internal issue here: https://github.com/jasp-stats/INTERNAL-jasp/issues/2421
It may be completed for version 0.20 or later.
I will mark this issue as duplicate but keep it open for reference.

[stvnbird]

A previous post indicates this is a niche issue. Just wanted to mention that data sets with more than 1000 columns is very common in the social sciences (sociology, criminology, political science, etc.)--basically any field that uses large scale surveying. Perhaps that is not an intended target audience for JASP, though. Thank you for the great work you do here!

[tomtomme]

This should be fixed with jasp-stats/jasp-desktop#5636
It will be hopefully released in some weeks with version 0.19.1
at least 16k columns should load fine then

[JorisGoosen]

0.19.1 is going to be the hotfix right?
This is not in there.

@boutinb @RensDofferhoff

[JeffS1984]

Still not fixed with 19 -- version 17.3 works for my purposes but hoping the next version has it fixed.

…

On Fri, Sep 6, 2024 at 11:16 AM Joris Goosen ***@***.***> wrote: 0.19.1 is going to be the hotfix right? This is not in there. @boutinb <https://github.com/boutinb> @RensDofferhoff <https://github.com/RensDofferhoff> — Reply to this email directly, view it on GitHub <#2327 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVQYHV2BWRSZIBGLXZDXVD3ZVHBLPAVCNFSM6AAAAAA5U463SKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZUGI4DCNJWGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RensDofferhoff]

@JeffS1984 You can download one of our nightlies marked 'development'.
These have the fixes applied https://static.jasp-stats.org/Nightlies/

@JorisGoosen We could include it? it does not interfere with Rpkg state?