[Xerte-dev] Re: example_zipper issue revisited

Thu May 1 00:04:38 BST 2014

Anyone still awake? Anyone able to test a new export zip script?

Unzip and drop it in /modules/xerte/

Uses the unix command zip and pipes the stream back into PHP in 8192 chunks

Not sure it will work any better with large files but I tested from a Linux server with a 79 Mb zip without changing my settings (not sure if they are at 128Mb but wouldn't have thought they would be more than that...

Also, might not work on Windows...

Regards,

John Smith | Learning Technologist
Room A251, Govan Mbeki Building | School of Health & Life Sciences | Glasgow Caledonian University
Cowcaddens Road | Glasgow | G4 0BA

Please address ALL support requests to hlsblt at gcu.ac.uk where one of the School Learning Technologists will pick up the job. This will ensure that all jobs are completed as promptly as possible.
________________________________________
From: xerte-dev-bounces at lists.nottingham.ac.uk [xerte-dev-bounces at lists.nottingham.ac.uk] On Behalf Of Pat L (pgogy) [xerte at pgogywebstuff.com]
Sent: 30 April 2014 22:04
To: For Xerte technical developers
Subject: [Xerte-dev] Re: example_zipper issue revisited

62MB is a big MP4 - especially given the limits on display
128 might work, but memory_limit and max_execution_time problems are dependent on uploads, so it is more likely to work, but not a guarantee - so "requirement" is perhaps not enough

On Wed, Apr 30, 2014 at 9:16 PM, Ron Mitchell <ronm at mitchellmedia.co.uk<mailto:ronm at mitchellmedia.co.uk>> wrote:
I don't think an alert is displayed to the user when it fails and the example_zipper files are left in the project folder and I think increase after each failed attempt.

I know on the one hand it's not best practice to upload mp4's direct and better to use a streaming server but there are obviously occasions where people do need to upload direct and 62MB as an example isn't exactly huge.

Sounds like non-default memory_limit or at least increase from 128MB has become a system requirement?

From: xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk> [mailto:xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk>] On Behalf Of Pat L (pgogy)
Sent: 30 April 2014 21:03

To: For Xerte technical developers
Subject: [Xerte-dev] Re: example_zipper issue revisited

It's not the code, it's the file sizes.
MP4s will be bigger when zipped than they are as files
Example zipper is the temp file name the code uses, i assume when it runs out of memory it dumps the file.
There is a logic to doing a file size check before exporting and warning with large files

On Wed, Apr 30, 2014 at 7:58 PM, Ron Mitchell <ronm at mitchellmedia.co.uk<mailto:ronm at mitchellmedia.co.uk>> wrote:
Yep it's in 2.1 https://github.com/thexerteproject/xerteonlinetoolkits/blob/2.1/modules/xerte/export.php line 61<https://github.com/thexerteproject/xerteonlinetoolkits/blob/2.1/modules/xerte/export.php%20line%2061>

and in develop too:  https://github.com/thexerteproject/xerteonlinetoolkits/blob/develop/modules/xerte/export.php line 61<https://github.com/thexerteproject/xerteonlinetoolkits/blob/develop/modules/xerte/export.php%20line%2061>

I've been testing further on a server where I can change anything and it does seem like increasing memory_limit resolves the issue but I'm not sure what the equation would be for file size v required limit or if any of the other php settings have a part to play too. I can't be sure but this seems like a relatively new issue too.  My tests so far…

1. Small projects don't seem to be a problem - no example_zipper created and projects export ok.

2. I was alerted to the problem again by a user of the Techdis sandpit who reported not being able to export here project yet was able to previously. The particular project included a 62MB mp4 so that's what I've been testing with now but on a different server. On this server with the memory_limit set to 128MB example_zipper files were created and part of the exported zip as well as left on the server.

3. Increasing the memory_limit to 300MB enable the export to work and no example_zipper created.

4. I mentioned I had a similar issue with a Wordpress install recently where the server host had to increase the memory_limit and in that case to 256MB and that seems to be a regular problem especially for people on shared hosting where they can't change the memory limit themselves. 128MB seems to be the default but I guess it could vary quite a bit depending on hosts.

5. What I'm not sure about is if this is a new problem due to changes in Xerte code and whether it's been fixed previously as I think John suggested.?

6. There are so many variables here I can't test reliably without others testing too e.g. I don't know if the combined file size with lots of smaller files gives the same  result as this single 62MB file or if other PHP settings also have a part to play. My fear is that we don't know how big the problem is. Partly because export isn't as common as just creating and linking and that together with project size variation, settings variation etc and perhaps some not knowing that these files even exist means it could be quite common without many knowing.

Ron

From: xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk> [mailto:xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk>] On Behalf Of Pat L (pgogy)
Sent: 30 April 2014 18:53
To: For Xerte technical developers
Subject: [Xerte-dev] Re: example_zipper issue revisited

Does exampler_zipper appear in the code, it sounds like something I left in.

I imagine it is a PHP timeout issue kicking in, or Memory being maxed out.

The zip script it supposed to do a clean up. Maybe do it before export now as well

On Wed, Apr 30, 2014 at 11:02 AM, Ron Mitchell <ronm at mitchellmedia.co.uk<mailto:ronm at mitchellmedia.co.uk>> wrote:
Hi all
I've updated some installations using the current 2.1 git download and had
hoped that the example_zipper issue I mentioned previously would be fixed
but it seems it hasn't? However it might only be showing up when large files
are included or with certain php or server configurations etc or perhaps
it's been fixed in develop but not applied to 2.1? This may also be an issue
going unoticed to some extent and leaving lots of redundant zip files in
project folders. So testing and questions...

What I'm finding with current 2.1 code on more than one server is that
exporting smaller projects works fine with no example_zipper files left on
the server or included in the zip download. However if a project includes
larger media files it either doesn't export or when it does it leaves
example_zipper files in the project folder as well as including them in the
zip download. I'm not sure where small - large overlaps but for instance
I've been testing export of a project containing a 60MB mp4.

Q1. Was this a known issue fixed in develop but not in current 2.1 branch?

Q2. Could this only be occurring with particular PHP/server settings and if
so what are the relevant settings?

Q3. Can others reproduce this using current 2.1 branch?

Cheers
Ron

-----Original Message-----
From: xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk>
[mailto:xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk>] On Behalf Of Smith, John
Sent: 08 April 2014 14:05
To: For Xerte technical developers
Subject: [Xerte-dev] Re: session issue + example_zipper issue

I've not really seen any session issues on a windows install and on xampp
mainly though. Have never seen the invalid templat id error either...

the example_zipper_new... file seems to be related to old export code
(https://code.google.com/p/xerteonlinetoolkits/source/browse/trunk/modules/x
erte/export.php?spec=svn696&r=696) with _xyz on the end nothing shows in
Google but remove _xyz and you get hits...

Regards,

John Smith | Learning Technologist
Room A251, Govan Mbeki Building | School of Health & Life Sciences | Glasgow
Caledonian University Cowcaddens Road | Glasgow | G4 0BA
________________________________________
From: xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk>
[xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk>] On Behalf Of Julian Tenney
[Julian.Tenney at nottingham.ac.uk<mailto:Julian.Tenney at nottingham.ac.uk>]
Sent: 08 April 2014 13:39
To: For Xerte technical developers
Subject: [Xerte-dev] Re: session issue + example_zipper issue

Never seen 'invalid template id'. But 'session ID not set' is doing my head
in.

I don't recognise 2.

From: xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk>
[mailto:xerte-dev-bounces at lists.nottingham.ac.uk<mailto:xerte-dev-bounces at lists.nottingham.ac.uk>] On Behalf Of Ron Mitchell
Sent: 08 April 2014 13:23
To: 'For Xerte technical developers'
Subject: [Xerte-dev] session issue + example_zipper issue

Hi all
I think an issue I mentioned in the thread entitled testing results got lost
in the discussion about re-running setup so I'm checking again along with a
separate issue I've noticed.

Issue 1
I downloaded xertetoolkits_2.1 Version:2.1 Last update  2014-03-14 from the
community site and from my testing this revision is not reliable e.g. one
minute I can create an LO and the next minute I get one of two errors when
trying to create a new lo:
Session ID not set
or
Invalid template_id (could not find in DB)

Q. Does this not happen for others?

Issue 2
In an older revision that I've been using which seems to be relatively
trouble free I've noticed that one or more files named something like
example_zipper_new_xyz. is getting created and included in any successful
export but also still residing in the project folder on the server.

Q. Does anyone recognise this? Was it fixed in a later revision that doesn't
have the session issue above?

Cheers
Ron

This message and any attachment are intended solely for the addressee and
may contain confidential information. If you have received this message in
error, please send it back to me, and immediately delete it.   Please do not
use, copy or disclose the information contained in this message or in any
attachment.  Any views or opinions expressed by the author of this email do
not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system,
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

Glasgow Caledonian University is a registered Scottish charity, number
SC021474

Winner: Times Higher Education's Widening Participation Initiative of the
Year 2009 and Herald Society's Education Initiative of the Year 2009.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en
.html

Winner: Times Higher Education's Outstanding Support for Early Career
Researchers of the Year 2010, GCU as a lead with Universities Scotland
partners.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,e
n.html

_______________________________________________
Xerte-dev mailing list
Xerte-dev at lists.nottingham.ac.uk<mailto:Xerte-dev at lists.nottingham.ac.uk>
http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
This message and any attachment are intended solely for the addressee and
may contain confidential information. If you have received this message in
error, please send it back to me, and immediately delete it.   Please do not
use, copy or disclose the information contained in this message or in any
attachment.  Any views or opinions expressed by the author of this email do
not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system,
you are advised to perform your own checks. Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

_______________________________________________
Xerte-dev mailing list
Xerte-dev at lists.nottingham.ac.uk<mailto:Xerte-dev at lists.nottingham.ac.uk>
http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev
This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it.   Please do not use, copy or disclose the information contained in this message or in any attachment.  Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment
may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.

This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it.   Please do not use, copy or disclose the information contained in this message or in any attachment.  Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.

This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.

_______________________________________________
Xerte-dev mailing list
Xerte-dev at lists.nottingham.ac.uk<mailto:Xerte-dev at lists.nottingham.ac.uk>
http://lists.nottingham.ac.uk/mailman/listinfo/xerte-dev

Glasgow Caledonian University is a registered Scottish charity, number SC021474
-------------- next part --------------
A non-text attachment was scrubbed...
Name: export.zip
Type: application/x-zip-compressed
Size: 2710 bytes
Desc: export.zip
URL: <http://lists.nottingham.ac.uk/pipermail/xerte-dev/attachments/20140501/280ca8e6/attachment.bin>