我需要在设计上载反馈的文件和延迟处理使用heroku上,回形针,延迟作业,如果有必要,S3。它的部分在其他地方进行了讨论,但我无法找到一个完整讨论的任何地方。
I need feedback on the design for uploading and delayed processing of a file using heroku, paperclip, delayed job and, if necessary, s3. Parts of it have been discussed in other places but I couldn't find a complete discussion anywhere.
任务描述:
在上传文件(使用回形针S3在Heroku / DB)。文件需要私人,因为它包含敏感数据。 在处理队列文件(延迟工作) 在工作中获取队列中运行 在文件被检索(从S3 / DB),并处理完毕 在文件被删除(从S3 / DB)由于我使用的延迟的工作,我有存储文件在数据库或S3之间作出选择。我假设存储在Web服务器上的文件出了问题,因为我使用Heroku和延误的工作。上传文件到S3需要很长的时间。但是,在分贝存储文件是更昂贵。理想情况下,我们希望将处理为尽快完成。
Since I am using delayed job, I have to decide between storing the file in the database or on s3. I am assuming that storing the file on the web server is out of the question as I am using heroku and delayed job. Uploading files to s3 takes a long time. But, storing files in db is more expensive. Ideally, we would want the processing to finish as quickly as possible.
什么是比较常见的设计模式?在S3上存储的文件?在数据库存储文件?存储在S3用于检索任何特别的推荐宝石和流程文件(AWS-S3?S3?)?
What is the more common design pattern? Store files on s3? Store files in db? Any particular recommended gems used to retrieve and process files stored in s3 (aws-s3? s3?)?
Heroku的为30秒的任一服务器的请求超时(深知这一点),所以肯定存储在S3上的文件是必须的。
Heroku has a timeout of 30 seconds on any server request (learnt the hard way), so definitely storing files on s3 is a must.
尝试 carrierwave (的 carrierwave railscasts),而不是回形针,因为我preFER增加的助手说加入进来,再加上有一些伟大的插件,像的 carrierwave_direct 上传大文件到S3,它与carrierwave很好地集成。
Try carrierwave (carrierwave railscasts) instead of paperclip, as I prefer the added helpers that come onboard, plus there a number of great plugins, like carrierwave_direct for uploading large files to s3, which integrate nicely with carrierwave.
的delayed_job (railscasts - 的 delayed_job的)会很好地工作从S3和可能需要的任何其他后台处理删除文件。
Delayed_job (railscasts - delayed_job) will work nicely for deleting files from s3 and any other background processing that may be required.
我的宝石文件包括以下内容:
My gem file includes the following:
gem 'delayed_job'
gem "aws-s3", :require => 'aws/s3'
gem 'fog'
gem 'carrierwave'
gem 'carrierwave_direct'
雾创业板是一个很好的方式,在一个地方所有的帐户信息,并设置了一切相当不错。对于AWS创业板如何做的,良好的资源。
fog gem is a nice way to have all your account info in a single place and sets up everything quite nicely. For the AWS gem how-to, good resource.
下面是一个简单的控制器提交表单时上传(肯定有这样做的更好的方法,但为了说明)
Here is a sample controller when submitting a form to upload (there are definitely better ways of doing this, but for illustrative purposes)
def create
@asset = Asset.new(:description => params[:description], :user_id => session[:id], :question_id => @question.id)
if @asset.save && @asset.update_attributes(:file_name => sanitize_filename(params[:uploadfile].original_filename, @asset.id))
AWS::S3::S3Object.store(sanitize_filename(params[:uploadfile].original_filename, @asset.id), params[:uploadfile].read, 'bucket_name', :access => :private, :content_type => params[:uploadfile].content_type)
if object.content_length.to_i < @question.emailatt.to_i.megabytes && object.content_length.to_i < 5.megabytes
url = AWS::S3::S3Object.url_for(sanitize_filename(params[:uploadfile].original_filename, @asset.id), 'bucket_name')
if @asset.update_attributes(:download_link => 1)
if Usermailer.delay({:run_at => 5.minutes.from_now}).attachment_user_mailer_download_notification(@asset, @question)
process_attachment_user_mailer_download(params[:uploadfile], @asset.id, 24.hours.from_now, @question.id)
flash[:notice] = "Thank you for the upload, we will notify this posts author"
end
end
end
else
@asset.destroy
flash[:notice] = "There was an error in processing your upload, please try again"
redirect_to(:controller => "questions", :action => "show", :id => @question.id)
end
end
private
def sanitize_filename(file_name, id)
just_filename = File.basename(file_name)
just_filename.sub(/[^\w\.\-]/,'_')
new_id = id.to_s
new_filename = "#{new_id}" + just_filename
end
def delete_process(uploadfile, asset_id, time, question_id)
asset = Asset.find(:first, :conditions => ["id = ?", asset_id])
if delete_file(uploadfile, asset_id, time) && asset.destroy
redirect_to(:controller => "questions", :action => "show", :id => question_id)
end
end
def process_attachment_user_mailer_download(uploadfile, asset_id, time, question_id)
asset = Asset.find(:first, :conditions => ["id = ?", asset_id])
if delete_file(uploadfile, asset_id, time) && @asset.delay({:run_at => time}).update_attributes(:download_link => 0)
redirect_to(:controller => "questions", :action => "show", :id => question_id)
end
end
#S3 METHODS FOR CREATE ACTION
#deletes the uploaded file from s3
def delete_file(uploadfile, asset_id, time)
AWS::S3::S3Object.delay({:run_at => time}).delete(sanitize_filename(uploadfile.original_filename, asset_id), 'bucket_name')
end
很多不必要的code,我知道(写这个的时候,我开始使用Rails)。希望这会给参与编写此类应用程序的过程中的一些想法。希望它能帮助。