It is currently Fri May 26, 2017 7:24 pm



Welcome
Welcome to rfobasic

You are currently viewing our boards as a guest, which gives you limited access to view most discussions and access our other features. By joining our free community, you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content, and access many other special features. In addition, registered members also see less advertisements. Registration is fast, simple, and absolutely free, so please, join our community today. **You are not required to provide truthful information to any registration questions. Be whomever you wish to be.!


Post new topic Reply to topic  [ 42 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next
Author Message
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 3:00 am 
Offline
User avatar

Joined: Tue Jan 03, 2012 9:31 am
Posts: 5518
Location: Paris, France
Gikam: commands do not act differently, some of them just have a fallback in case the file passed as argument doesn't exist on sdcard.

Marc: I agree, the manual is already very heavy as it is, I can't imagine aadding considerations on apk mode in every paragraph (or more precisely every command that has a specific apk behavior).
The final user shouldn't have to bother about apk mode, it should be transparent, i.e. what he does in rfo-basic/data in Editor mode, he uses the exact same commands i.e. the same .bas program to do in assets/<my_app>/data in apk mode if he correctly added the resources in the apk.

...which puts me in a pickle... Because I really cannot make Zip.Dir and Zip.Read commands work in apk mode, except by re-writing them entirely and make them very different from the commands in Editor mode, meaning doubling every method I wrote (ugly and heavy).

Because at the beginning, I used an inputstream as the zip fileInfo 'reader' but with it it is impossible to access randomly the zip entries: after my Zip.Open command, I could only read them sequentially, and this always put me at the end of the zip file after 'seeking' an entry, so I would have needed to close the stream and re-create / re-open it at each command invocation.

I replaced the inputstream with a ZipFile which allows a random access, and not closing the 'reader' after you reach the end of the zip file. But ZipFile do not work with assets as is shown in the StackOverflows I mentioned above...

Any idea or advice?

_________________
- Creator of the Android BASIC! Compiler


Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 3:37 am 
Offline
User avatar

Joined: Thu Jan 08, 2015 11:28 am
Posts: 1088
Location: .NET
mougino wrote:
Gikam: commands do not act differently, some of them just have a fallback in case the file passed as argument doesn't exist on sdcard.

Marc: I agree, the manual is already very heavy as it is, I can't imagine aadding considerations on apk mode in every paragraph (or more precisely every command that has a specific apk behavior).
The final user shouldn't have to bother about apk mode, it should be transparent, i.e. what he does in rfo-basic/data in Editor mode, he uses the exact same commands i.e. the same .bas program to do in assets/<my_app>/data in apk mode if he correctly added the resources in the apk.

...which puts me in a pickle... Because I really cannot make Zip.Dir and Zip.Read commands work in apk mode, except by re-writing them entirely and make them very different from the commands in Editor mode, meaning doubling every method I wrote (ugly and heavy).

Because at the beginning, I used an inputstream as the zip fileInfo 'reader' but with it it is impossible to access randomly the zip entries: after my Zip.Open command, I could only read them sequentially, and this always put me at the end of the zip file after 'seeking' an entry, so I would have needed to close the stream and re-create / re-open it at each command invocation.

I replaced the inputstream with a ZipFile which allows a random access, and not closing the 'reader' after you reach the end of the zip file. But ZipFile do not work with assets as is shown in the StackOverflows I mentioned above...

Any idea or advice?



how about 2 different read commands - one for sequential one for random? afterall we have text.read and byte.read
or use the idea about decompressing the zip elsewhere first, like a temp directory. http://stackoverflow.com/questions/3425 ... in-android

_________________
https://github.com/evolbug
http://toobasic.jimdo.com


Last edited by evolbug on Wed Jun 29, 2016 3:42 am, edited 1 time in total.

Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 3:42 am 
Offline
User avatar

Joined: Tue Jan 03, 2012 9:31 am
Posts: 5518
Location: Paris, France
I don't see the link :?

[edit] and no, even with 2 functions, in Editor the 2 would work but in Apk only 1 would work, so that's not acceptable

_________________
- Creator of the Android BASIC! Compiler


Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 3:44 am 
Offline
User avatar

Joined: Thu Jan 08, 2015 11:28 am
Posts: 1088
Location: .NET
mougino wrote:
I don't see the link :?

[edit] and no, even with 2 functions, in Editor the 2 would work but in Apk only 1 would work, so that's not acceptable


but you can detect when you are in an apk right? so you could replace the method with an apk-compliant one

_________________
https://github.com/evolbug
http://toobasic.jimdo.com


Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 3:51 am 
Offline
User avatar

Joined: Tue Jan 03, 2012 9:31 am
Posts: 5518
Location: Paris, France
Yes, I think I will follow your second advice, redirect the stream to a temp dir or a File or the like, but this should impact the performances.

Nicolas

_________________
- Creator of the Android BASIC! Compiler


Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 3:56 am 
Offline
User avatar

Joined: Thu Jan 08, 2015 11:28 am
Posts: 1088
Location: .NET
mougino wrote:
Yes, I think I will follow your second advice, redirect the stream to a temp dir or a File or the like, but this should impact the performances.

Nicolas


it's a start atleast. also you could maybe copy the files out of assets into private space on install? or on first run? that would somewhat improve performance instead of TEMPing it each time it runs

OR actually i have an idea. if the program is in an apk - extract the assets into private space on first run/command call, then check if it exists on subsequent runs/calls, redirecting control to normal file operations

_________________
https://github.com/evolbug
http://toobasic.jimdo.com


Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 4:02 am 
Offline
User avatar

Joined: Tue Jan 03, 2012 9:31 am
Posts: 5518
Location: Paris, France
Hi evo, I didn't mention it but I also want to respect the existing BASIC! architecture :)
I'm not alone on this, so I won't make big architectural changes like dumping the assets to private space and access them from there.
I'll try to copy the existing behavior for Byte.* and Text.* which is to deal with inputstreams.

Nicolas

_________________
- Creator of the Android BASIC! Compiler


Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 6:53 am 
Offline

Joined: Wed Oct 03, 2012 9:53 am
Posts: 2797
Location: Colorado, U.S.
I would be concerned about dropping a big file into space the user can't see. Suddenly there is less SDcard space, where did it go? And how do I get it back?

If I understand you, Nicolas, you're saying you can do things with a ZipFile object that you can't do with a ZipInputStream. If you want both files and assets to work the same way, you'd have to use the ZipInputStream, which does not give you random access, so it's slower. But your experiment to make reading faster used a BufferedInputStream, also losing random access. Am I mixing up two cases?


Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 8:17 am 
Offline
User avatar

Joined: Tue Jan 03, 2012 9:31 am
Posts: 5518
Location: Paris, France
Hi Marc, ZipInputStream is not only slower, it simply does not work with the Zip.read command.

The way Byte.read.* and Text.readln are coded in BASIC! is that you use an input stream, stored in a FileInfo, that was created at the time of Byte.open r and Text.open r respectively (these input stream are respectively called mByteReader and mTextReader).

Important point is that these input streams are not closed until you call Byte.close and Text.close respectively.

Situation: imagine a zip composed of the following 3 files:
Code:
file1.ext
file2.ext
file3.ext

If I use for the Zip.open and Zip.read commands a ZipInputStream - in the same way I use other input streams for Byte.* and Text.* - and if I do a Zip.read fid, buf$, "file3.ext" then I will not be able to do a Zip.read of "file1.ext" nor "file2.ext" later on. The only way to achieve this would be to close the ZipInputStream stored in mZipReader and re-create a new ZipInputStream (17 lines of code in executeZIP_OPEN()).

Because the only method that a ZipInputStream offers is .getNextEntry(), there is no getEntry(entryName) with ZipInputStream.

So this breaks the similarity with the other Byte.* and Text.* methods, where it is possible to navigate in the stream without closing it and re-creating it each time, and I think doing that for Zip.* is not acceptable (I have the feeling that closing and re-creating the input stream each time you access an entry in the zip will be very resource consuming).

ZipFile allows to fix that, it is more flexible than ZipInputStream in that it allows random access without closing the ZipFile object, you can read this article that explains the difference between them.

Still, once you have an entry randomly accessed through ZipFile.getEntry(entryName), you need an input stream to unzip it ;) there's a method getInputStream() that gives you access to an input stream of the entry itself (not of the global zip like ZipInputStream does). This is where I made the speed improvements, on this getInputStream(ZipFile.getEntry(entryName)).

Back to the management of Zip.open / Zip.read in apk mode, I decided to implement a copy to the protected space and create a ZipFile from there. This is not perfect but it works well (and I check if the file is still there on second and following accesses).

Marc, do you want I send you the files (Run.java and Basic.java), rather than committing on GitHub ? And I leave it up to you to implement a better alternative ? (sorry ^^ that's what we call a poisoned gift in french!)

Nicolas

_________________
- Creator of the Android BASIC! Compiler


Report this post
Top
 Profile  
 
 Post subject: Re: Zip commands
Unread postPosted: Wed Jun 29, 2016 1:11 pm 
Offline

Joined: Wed Oct 03, 2012 9:53 am
Posts: 2797
Location: Colorado, U.S.
Ah, I see. I assumed getting a ZipInputStream on a resource gave you the whole resource. It does not.
It seems strange to me that a ZipInputStream can't skip to the next entry without reading (and decompressing) the stream, but that's how it is.
Very different from the other streams we use. Disappointing.

I don't like that the same program will run faster from the Editor than as a standalone apk. On the other hand, many apps are already like that. If you, the programmer, know you will run from an apk, you may copy a state file from assets to SDcard on first run, and find it still there on subsequent runs. One hopes that a programmer making such an apk for other users tells the users that a file gets created, where it is, and how to manage it.

I don't like trying to manage an asset as a file without the user's knowledge or control. No -- I said that wrong. Lots of apps create files that users can't see, like high-score files, or files that should be deleted when the app is uninstalled. That's okay for programs. It is not okay for programming languages. At least not the kind of language this is. We've generally avoided doing things in BASIC! that make persistent changes in the system, other than the Preferences file.

We've talked about providing a way for a BASIC! program to write to private storage; this is just another example to show we need it. App-developers can choose one technique, but language-developers must provide many, and should avoid favoring any.

I think that if we can't use zipped assets without creating a file, then ZIP commands should operate only on real files. If you want to use a zip file from assets, your program would first have to use non-ZIP commands to copy it to the sdcard. Programmers control their own storage, and the BASIC! interpreter code is simpler. Is this acceptable?

- Marc


Report this post
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 42 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next


Who is online

Users browsing this forum: No registered users and 0 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
suspicion-preferred