[Xerte-dev] Re: example_zipper issue revisited

Thu May 1 11:16:45 BST 2014

Hi Ron,

I don't know if it's worth keeping testing different eventualities until we have a better idea of what is going on. The problem with these zipper files is that each time a new one is created and not deleted then it will be included in further zip attempts because it shared the folder with the copied files. I did read somewhere that the zip command that I am using ignores files/folders begin with a . (dot) and perhaps the php library can be made to do similar which would at least keep the temporary zipper folders out of the downloaded zip.

I'm a bit busy this morning but will in the afternoon try to test how the inclusion of different sized files correlates to memory limits and see if I can predict whether it will crash the script like the example that I based on your test case.

Regards,

John Smith | Learning Technologist
Room A251, Govan Mbeki Building | School of Health & Life Sciences | Glasgow Caledonian University
Cowcaddens Road | Glasgow | G4 0BA

Please address ALL support requests to hlsblt at gcu.ac.uk where one of the School Learning Technologists will pick up the job. This will ensure that all jobs are completed as promptly as possible.
________________________________________
From: xerte-dev-bounces at lists.nottingham.ac.uk [xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Ron Mitchell [ronm at mitchellmedia.co.uk]
Sent: 01 May 2014 10:58
To: 'For Xerte technical developers'
Subject: [Xerte-dev] Re: example_zipper issue revisited

Hi John
the problem is when I originally raised this issue on 8th April it coincided
with discovering loads of these un-removed example_zipper files on the
sandpit install. At that time I also mentioned the separate session issue as
a reason I hadn't updated to a newer revision and I think you mentioned that
you thought Tom had fixed this zipper issue in a later revision. So once
we'd identified the cause of the session issue (or at least Julian's attempt
at fixing his FF issue) I updated the sandpit code as well as other servers.
Now I've been using the 2.1 branch for this rather than develop so I'm not
sure if Tom's fix is in develop but not 2.1 but I think the code is the same
at the moment.

In short although I've been using a 62MB file from a project a user was
unable to export, previously there were loads of these zipper files in loads
of different user/project folders and there are probably quite a few again
now prior to my changing the memory limit.

I could try resetting the memory_limit back on my own server and test again
but my previous tests with the 62MB file certainly weren't crashing the
server. Let me know if you think its worth testing with larger files and the
default 128MB limit or if your and David's later responses have ruled that
out as being worthwhile?

I know Tom said last night that he was going to revisit his fix for this so
would be good to hear back from Tom once he's done that.

Cheers
Ron

-----Original Message-----
From: xerte-dev-bounces at lists.nottingham.ac.uk
[mailto:xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Smith, John
Sent: 01 May 2014 09:57
To: For Xerte technical developers
Subject: [Xerte-dev] Re: example_zipper issue revisited

Hi Ron,

Have been thinking about your issue and just wondering whether you might
have inadvertently chosen the worst file size possible for the video with
128Mb memory available to PHP. Let me explain (and this is entirely
hypothetical)...

The zipper creates a blank object (<0.1Mb memory used) The library loads all
the other files individually and zips them into the object (0Mb ->
compressed size of all js/html is perhaps around 1-2Mb or under) The library
then loads the mp4 (now it has the object = 2Mb and the video = 62Mb in
memory) It adds it to the zip object but it's also still in memory (mp4
doesn't compress at all so we have 2 + 62 + 62 =126Mb which is less than
128Mb) It then closes the connection to the mp4 and destroys the object
(memory now = 64Mb for the zip)

I'm wondering whether a 64 Mb or slightly bigger will cause you to be able
to crash that server too. Is that something you could try?

Regards,

John Smith | Learning Technologist
Room A251, Govan Mbeki Building | School of Health & Life Sciences | Glasgow
Caledonian University Cowcaddens Road | Glasgow | G4 0BA

Please address ALL support requests to hlsblt at gcu.ac.uk where one of the
School Learning Technologists will pick up the job. This will ensure that
all jobs are completed as promptly as possible.
________________________________________
From: xerte-dev-bounces at lists.nottingham.ac.uk
[xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Ron Mitchell
[ronm at mitchellmedia.co.uk]
Sent: 01 May 2014 09:09
To: 'For Xerte technical developers'
Subject: [Xerte-dev] Re: example_zipper issue revisited

Hi John
I was just going to test your export code on a server where I don't have
access to change php settings so the memory limit is fixed at 128MB. I know
what David means that ideally we want cross platform code but as you say
there are inevitably different solutions/restrictions for some e.g. no
control over memory limit. In fact the upload limit on this particular
shared server is set at the default 2MB so I had to upload the test 62MB mp4
via ftp. However I'm now confused further because this is the same XOT code
as the other servers I was using for testing so I expected the export to
fail when testing before replacing the export code with yours and yet it
didn't fail and I was able to download the zip which didn't contain any
example_zipper files.

I'm glad you're able to reproduce the issue but not sure how a server with
128MB memory limit is able to export ok without a change of code or memory
limit :-(

-----Original Message-----
From: xerte-dev-bounces at lists.nottingham.ac.uk
[mailto:xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Smith, John
Sent: 01 May 2014 08:38
To: xerte-dev at lists.nottingham.ac.uk
Subject: [Xerte-dev] Re: example_zipper issue revisited

All i know Dave is that, with the commented code uncommented, my system
couldn't download a 56Mb an 79Mb zip but with this code 79Mb came down no
problem.

We may have to use what we have access to on different systems. There may be
a similar way to pipe the result into windows or we fall back to the old
code and crank the memory requirements up yo 300Mb which not everyone can
most likely do anyway...

Regards

John Smith
Learning Technologist
School of Health and Life Sciences

Sent from Samsung Galaxy SII

David Goodwin <david at palepurple.co.uk> wrote:

<snip>
Uses the unix command zip and pipes the stream back into PHP in 8192 chunks

<snip>
Also, might not work on Windows...

So therefore it's not worth using?

Technically there is no benefit to doing the fread(....) over a
fpassthru(...) fpassthru does something like 8192 chunks internally.

David.

--
--
Pale Purple Ltd
http://www.palepurple.co.uk @PalePurpleLtd
0845 0046746 / 07792 380669
"Bespoke web and mobile application development"

This message and any attachment are intended solely for the addressee and
may contain confidential information. If you have received this message in
error, please send it back to me, and immediately delete it.   Please do not
use, copy or disclose the information contained in this message or in any
attachment.  Any views or opinions expressed by the author of this email do
not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system,
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Glasgow Caledonian University is a registered Scottish charity, number
SC021474

_______________________________________________
Xerte-dev mailing list
Xerte-dev at lists.nottingham.ac.uk
http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
This message and any attachment are intended solely for the addressee and
may contain confidential information. If you have received this message in
error, please send it back to me, and immediately delete it.   Please do not
use, copy or disclose the information contained in this message or in any
attachment.  Any views or opinions expressed by the author of this email do
not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system,
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

_______________________________________________
Xerte-dev mailing list
Xerte-dev at lists.nottingham.ac.uk
http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
This message and any attachment are intended solely for the addressee and
may contain confidential information. If you have received this message in
error, please send it back to me, and immediately delete it.   Please do not
use, copy or disclose the information contained in this message or in any
attachment.  Any views or opinions expressed by the author of this email do
not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system,
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Glasgow Caledonian University is a registered Scottish charity, number
SC021474

_______________________________________________
Xerte-dev mailing list
Xerte-dev at lists.nottingham.ac.uk
http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
This message and any attachment are intended solely for the addressee and
may contain confidential information. If you have received this message in
error, please send it back to me, and immediately delete it.   Please do not
use, copy or disclose the information contained in this message or in any
attachment.  Any views or opinions expressed by the author of this email do
not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system,
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

_______________________________________________
Xerte-dev mailing list
Xerte-dev at lists.nottingham.ac.uk
http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it.   Please do not use, copy or disclose the information contained in this message or in any attachment.  Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.

Glasgow Caledonian University is a registered Scottish charity, number SC021474